[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:49:01 dist-test-slave-jsw9 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:49:01 dist-test-slave-jsw9 krb5kdc[399](info): set up 2 sockets
Jun 19 16:49:01 dist-test-slave-jsw9 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 16:49:02.903127 392 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:49:03 dist-test-slave-jsw9 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:49:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:49:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:49:03.050593 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45377
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45377
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:03.143947 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:03.144248 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:03.144285 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:03.145747 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:03.145812 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:03.145833 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:03.145849 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:03.145864 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:49:03.147802 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:03.148169 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:03.148551 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:49:03.154608 412 server_base.cc:1034] running on GCE node
W20260619 16:49:03.154556 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:03.154569 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:03.154585 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:03.155051 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:03.155416 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:03.156610 412 hybrid_clock.cc:648] HybridClock initialized: now 1781887743156588 us; error 42 us; skew 500 ppm
Jun 19 16:49:03 dist-test-slave-jsw9 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:03.159777 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:49:03.160455 412 webserver.cc:460] Webserver started at http://127.0.98.62:46409/ using document root <none> and password file <none>
I20260619 16:49:03.160732 412 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:03.160791 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:03.160938 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:03.162441 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "db65011efa664168aba13a2a8091ec26"
format_stamp: "Formatted at 2026-06-19 16:49:03 on dist-test-slave-jsw9"
server_key: "256d10656caa9f6caba15aab4205a17f"
server_key_iv: "3a628148a3b3a6eacc02b6654fd2a422"
server_key_version: "encryptionkey@0"
I20260619 16:49:03.162884 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "db65011efa664168aba13a2a8091ec26"
format_stamp: "Formatted at 2026-06-19 16:49:03 on dist-test-slave-jsw9"
server_key: "256d10656caa9f6caba15aab4205a17f"
server_key_iv: "3a628148a3b3a6eacc02b6654fd2a422"
server_key_version: "encryptionkey@0"
I20260619 16:49:03.164433 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 16:49:03.165417 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:03.165660 412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:03.165766 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "db65011efa664168aba13a2a8091ec26"
format_stamp: "Formatted at 2026-06-19 16:49:03 on dist-test-slave-jsw9"
server_key: "256d10656caa9f6caba15aab4205a17f"
server_key_iv: "3a628148a3b3a6eacc02b6654fd2a422"
server_key_version: "encryptionkey@0"
I20260619 16:49:03.165844 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:03.175911 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:03.178608 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:03.178790 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:03.183527 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45377
I20260619 16:49:03.183615 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45377 every 8 connection(s)
I20260619 16:49:03.184051 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:03.185135 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:03.187040 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 412
I20260619 16:49:03.187176 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:49:03.188035 392 external_mini_cluster.cc:1428] Setting key 0f473a4f4680b546818b7081682f8b55
I20260619 16:49:03.188756 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db65011efa664168aba13a2a8091ec26: Bootstrap starting.
I20260619 16:49:03.189553 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db65011efa664168aba13a2a8091ec26: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:03.189884 480 log.cc:824] T 00000000000000000000000000000000 P db65011efa664168aba13a2a8091ec26: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:03.190676 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db65011efa664168aba13a2a8091ec26: No bootstrap required, opened a new log
I20260619 16:49:03.192590 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P db65011efa664168aba13a2a8091ec26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db65011efa664168aba13a2a8091ec26" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45377 } }
I20260619 16:49:03.192801 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P db65011efa664168aba13a2a8091ec26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:03.192839 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P db65011efa664168aba13a2a8091ec26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db65011efa664168aba13a2a8091ec26, State: Initialized, Role: FOLLOWER
I20260619 16:49:03.192988 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P db65011efa664168aba13a2a8091ec26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db65011efa664168aba13a2a8091ec26" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45377 } }
I20260619 16:49:03.193073 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P db65011efa664168aba13a2a8091ec26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:03.193099 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P db65011efa664168aba13a2a8091ec26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:03.193150 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db65011efa664168aba13a2a8091ec26 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:03.194139 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P db65011efa664168aba13a2a8091ec26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db65011efa664168aba13a2a8091ec26" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45377 } }
I20260619 16:49:03.194270 480 leader_election.cc:302] T 00000000000000000000000000000000 P db65011efa664168aba13a2a8091ec26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db65011efa664168aba13a2a8091ec26; no voters:
I20260619 16:49:03.194448 480 leader_election.cc:288] T 00000000000000000000000000000000 P db65011efa664168aba13a2a8091ec26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:49:03.194509 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db65011efa664168aba13a2a8091ec26 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:03.194684 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P db65011efa664168aba13a2a8091ec26 [term 1 LEADER]: Becoming Leader. State: Replica: db65011efa664168aba13a2a8091ec26, State: Running, Role: LEADER
I20260619 16:49:03.194736 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P db65011efa664168aba13a2a8091ec26 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:03.194793 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P db65011efa664168aba13a2a8091ec26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db65011efa664168aba13a2a8091ec26" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45377 } }
I20260619 16:49:03.195205 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P db65011efa664168aba13a2a8091ec26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db65011efa664168aba13a2a8091ec26. Latest consensus state: current_term: 1 leader_uuid: "db65011efa664168aba13a2a8091ec26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db65011efa664168aba13a2a8091ec26" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45377 } } }
I20260619 16:49:03.195214 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P db65011efa664168aba13a2a8091ec26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db65011efa664168aba13a2a8091ec26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db65011efa664168aba13a2a8091ec26" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45377 } } }
I20260619 16:49:03.195361 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P db65011efa664168aba13a2a8091ec26 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:03.195289 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P db65011efa664168aba13a2a8091ec26 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:03.195667 493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:03.196249 493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:03.198290 493 catalog_manager.cc:1347] Generated new cluster ID: cb4f699b39864f2a9624f4ae60fd8970
I20260619 16:49:03.198348 493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 16:49:03 dist-test-slave-jsw9 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:03.210835 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:03.189410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53916 (local address 127.0.98.62:45377)
0619 16:49:03.189713 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:03.189727 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:03.189993 (+ 266us) server_negotiation.cc:407] Connection header received
0619 16:49:03.192537 (+ 2544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:03.192547 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:03.192622 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:03.192811 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:03.194780 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:03.196216 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:03.197688 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:03.198513 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:03.198635 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:03.204553 (+ 5918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:03.204560 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:03.204561 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:03.204580 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:03.207735 (+ 3155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:03.208399 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:03.208404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:03.208404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:03.208542 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:03.208847 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:03.208848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:03.208849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:03.209233 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:03.209413 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:03.210348 (+ 935us) server_negotiation.cc:299] Negotiation successful
0619 16:49:03.210520 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":149,"threads_started":1}
I20260619 16:49:03.212294 493 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:49:03.212860 493 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:03.220849 493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db65011efa664168aba13a2a8091ec26: Generated new TSK 0
I20260619 16:49:03.221123 493 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:49:03.258136 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45377
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:03.353489 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:03.353780 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:03.353811 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:03.355306 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:03.355363 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:03.355397 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:49:03.357092 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45377
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:03.357573 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:03.357872 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:49:03.363785 506 server_base.cc:1034] running on GCE node
W20260619 16:49:03.363752 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:03.363752 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:03.363754 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:03.364199 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:03.364530 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:03.365689 506 hybrid_clock.cc:648] HybridClock initialized: now 1781887743365673 us; error 33 us; skew 500 ppm
Jun 19 16:49:03 dist-test-slave-jsw9 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:03.368862 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:49:03.369486 506 webserver.cc:460] Webserver started at http://127.0.98.1:38731/ using document root <none> and password file <none>
I20260619 16:49:03.369721 506 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:03.369773 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:03.369915 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:03.371305 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-0/data/instance:
uuid: "e2f60701c768402a88d9ca755a05df35"
format_stamp: "Formatted at 2026-06-19 16:49:03 on dist-test-slave-jsw9"
server_key: "58c1108a22afbce198345017c11096b6"
server_key_iv: "c22eae95f86f46bf3fc3b575f0ad0559"
server_key_version: "encryptionkey@0"
I20260619 16:49:03.371717 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e2f60701c768402a88d9ca755a05df35"
format_stamp: "Formatted at 2026-06-19 16:49:03 on dist-test-slave-jsw9"
server_key: "58c1108a22afbce198345017c11096b6"
server_key_iv: "c22eae95f86f46bf3fc3b575f0ad0559"
server_key_version: "encryptionkey@0"
I20260619 16:49:03.373345 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 16:49:03.374339 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:03.374568 506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:03.374678 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-0/wal
uuid: "e2f60701c768402a88d9ca755a05df35"
format_stamp: "Formatted at 2026-06-19 16:49:03 on dist-test-slave-jsw9"
server_key: "58c1108a22afbce198345017c11096b6"
server_key_iv: "c22eae95f86f46bf3fc3b575f0ad0559"
server_key_version: "encryptionkey@0"
I20260619 16:49:03.374755 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:03.390285 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:03.393078 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:03.393256 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:03.393563 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:03.393949 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:03.393985 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:03.394019 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:03.394043 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:03.401149 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42399
I20260619 16:49:03.401193 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42399 every 8 connection(s)
I20260619 16:49:03.401717 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:49:03.405171 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 506
I20260619 16:49:03.405344 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:49:03.405520 392 external_mini_cluster.cc:1428] Setting key 72eb3aa0088596cbb21e7a3deb3abc9c
Jun 19 16:49:03 dist-test-slave-jsw9 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:03.413836 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:03.402748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59605 (local address 127.0.98.62:45377)
0619 16:49:03.402807 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:03.402809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:03.403030 (+ 221us) server_negotiation.cc:407] Connection header received
0619 16:49:03.403816 (+ 786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:03.403817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:03.403836 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:03.403905 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:03.405164 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:03.405868 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:03.407098 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:03.407332 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:03.407395 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:03.410327 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:03.410329 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:03.410330 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:03.410340 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:03.412115 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:03.412652 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:03.412653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:03.412654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:03.412736 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:03.413039 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:03.413041 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:03.413041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:03.413204 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:03.413275 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:03.413613 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 16:49:03.413719 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 16:49:03.413992 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:03.402658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45377 (local address 127.0.98.1:59605)
0619 16:49:03.402851 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:03.402903 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:03.403667 (+ 764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:03.404055 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:03.404061 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:03.404733 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:03.405028 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:03.405037 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:03.405966 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:03.405967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:03.407007 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:03.407010 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:03.407416 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:03.407418 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:03.407609 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:03.408311 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:03.408325 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:03.410177 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:03.412234 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:03.412243 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:03.412246 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:03.412566 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:03.412830 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:03.412831 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:03.412832 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:03.412970 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:03.413277 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:03.413290 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:03.413462 (+ 172us) client_negotiation.cc:769] Sending connection context
0619 16:49:03.413566 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 16:49:03.413734 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":88,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 16:49:03.414732 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45377
I20260619 16:49:03.414867 635 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:03.415091 635 heartbeater.cc:511] Master 127.0.98.62:45377 requested a full tablet report, sending...
I20260619 16:49:03.415840 444 ts_manager.cc:194] Registered new tserver with Master: e2f60701c768402a88d9ca755a05df35 (127.0.98.1:42399)
I20260619 16:49:03.416590 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59605
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:49:03.434355 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45377
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:03.532486 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:03.532769 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:03.532802 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:03.534343 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:03.534403 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:03.534438 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:49:03.536301 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45377
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:03.536602 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:03.536850 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:03.542727 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:03.542850 640 server_base.cc:1034] running on GCE node
W20260619 16:49:03.542726 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:03.542954 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:03.543191 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:03.543529 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:03.544703 640 hybrid_clock.cc:648] HybridClock initialized: now 1781887743544692 us; error 33 us; skew 500 ppm
Jun 19 16:49:03 dist-test-slave-jsw9 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:03.548112 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:49:03.548782 640 webserver.cc:460] Webserver started at http://127.0.98.2:43905/ using document root <none> and password file <none>
I20260619 16:49:03.549094 640 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:03.549167 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:03.549330 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:03.550674 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-1/data/instance:
uuid: "c6668e4404da4031a5d4eef32de6d3e0"
format_stamp: "Formatted at 2026-06-19 16:49:03 on dist-test-slave-jsw9"
server_key: "d021d3f97c8636abdf0bf6bc9f73e84e"
server_key_iv: "f1fa6a87199e6f5ed2558310dab751e6"
server_key_version: "encryptionkey@0"
I20260619 16:49:03.551008 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c6668e4404da4031a5d4eef32de6d3e0"
format_stamp: "Formatted at 2026-06-19 16:49:03 on dist-test-slave-jsw9"
server_key: "d021d3f97c8636abdf0bf6bc9f73e84e"
server_key_iv: "f1fa6a87199e6f5ed2558310dab751e6"
server_key_version: "encryptionkey@0"
I20260619 16:49:03.552611 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 16:49:03.553462 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:03.553666 640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:03.553764 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-1/wal
uuid: "c6668e4404da4031a5d4eef32de6d3e0"
format_stamp: "Formatted at 2026-06-19 16:49:03 on dist-test-slave-jsw9"
server_key: "d021d3f97c8636abdf0bf6bc9f73e84e"
server_key_iv: "f1fa6a87199e6f5ed2558310dab751e6"
server_key_version: "encryptionkey@0"
I20260619 16:49:03.553833 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:03.562987 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:03.565306 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:03.565436 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:03.565662 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:03.566012 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:03.566056 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:03.566087 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:03.566104 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:03.573596 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44815
I20260619 16:49:03.573693 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44815 every 8 connection(s)
I20260619 16:49:03.574075 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:49:03.580564 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 640
I20260619 16:49:03.580660 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:49:03.580837 392 external_mini_cluster.cc:1428] Setting key fa0bf9d356ac1c81f521dc96b559c264
Jun 19 16:49:03 dist-test-slave-jsw9 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:03.585080 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:03.575014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51095 (local address 127.0.98.62:45377)
0619 16:49:03.575100 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:03.575103 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:03.575352 (+ 249us) server_negotiation.cc:407] Connection header received
0619 16:49:03.575894 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:03.575895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:03.575907 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:03.575970 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:03.577104 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:03.577698 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:03.579063 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:03.579298 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:03.579364 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:03.581992 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:03.581994 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:03.581994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:03.582004 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:03.583525 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:03.584062 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:03.584063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:03.584064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:03.584135 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:03.584401 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:03.584402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:03.584402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:03.584536 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:03.584584 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:03.584923 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 16:49:03.585002 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:49:03.585356 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:03.574983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45377 (local address 127.0.98.2:51095)
0619 16:49:03.575160 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:03.575212 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:03.575800 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:03.576046 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:03.576051 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:03.576713 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:03.576983 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:03.576990 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:03.577821 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:03.577822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:03.578961 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:03.578964 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:03.579421 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:03.579422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:03.579572 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:03.580173 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:03.580183 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:03.581893 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:03.583623 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:03.583633 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:03.583636 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:03.583986 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:03.584196 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:03.584197 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:03.584197 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:03.584343 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:03.584586 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:03.584591 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:03.584782 (+ 191us) client_negotiation.cc:769] Sending connection context
0619 16:49:03.584905 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 16:49:03.585094 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":85,"threads_started":1}
I20260619 16:49:03.585897 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45377
I20260619 16:49:03.586028 769 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:03.586277 769 heartbeater.cc:511] Master 127.0.98.62:45377 requested a full tablet report, sending...
I20260619 16:49:03.586715 444 ts_manager.cc:194] Registered new tserver with Master: c6668e4404da4031a5d4eef32de6d3e0 (127.0.98.2:44815)
I20260619 16:49:03.587117 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51095
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:49:03.608678 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45377
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:03.704731 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:03.704999 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:03.705036 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:03.706413 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:03.706461 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:03.706501 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:49:03.708104 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45377
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:03.708398 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:03.708652 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:03.715142 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:03.715139 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:03.715144 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:03.715409 774 server_base.cc:1034] running on GCE node
I20260619 16:49:03.715667 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:03.715982 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:03.717139 774 hybrid_clock.cc:648] HybridClock initialized: now 1781887743717129 us; error 32 us; skew 500 ppm
Jun 19 16:49:03 dist-test-slave-jsw9 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:03.720635 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:49:03.721223 774 webserver.cc:460] Webserver started at http://127.0.98.3:35641/ using document root <none> and password file <none>
I20260619 16:49:03.721470 774 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:03.721519 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:03.721653 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:03.722918 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-2/data/instance:
uuid: "d334f60a413348acb4272e508f8276d9"
format_stamp: "Formatted at 2026-06-19 16:49:03 on dist-test-slave-jsw9"
server_key: "d7c60de6c3ec3052cfc0a76ed6cb8b99"
server_key_iv: "83d7cd85bbc5d1d631c28cb0c4214952"
server_key_version: "encryptionkey@0"
I20260619 16:49:03.723291 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d334f60a413348acb4272e508f8276d9"
format_stamp: "Formatted at 2026-06-19 16:49:03 on dist-test-slave-jsw9"
server_key: "d7c60de6c3ec3052cfc0a76ed6cb8b99"
server_key_iv: "83d7cd85bbc5d1d631c28cb0c4214952"
server_key_version: "encryptionkey@0"
I20260619 16:49:03.724784 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 16:49:03.725562 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:03.725708 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 16:49:03.725798 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-2/wal
uuid: "d334f60a413348acb4272e508f8276d9"
format_stamp: "Formatted at 2026-06-19 16:49:03 on dist-test-slave-jsw9"
server_key: "d7c60de6c3ec3052cfc0a76ed6cb8b99"
server_key_iv: "83d7cd85bbc5d1d631c28cb0c4214952"
server_key_version: "encryptionkey@0"
I20260619 16:49:03.725862 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:03.741044 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:03.743669 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:03.743839 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:03.744148 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:03.744562 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:03.744601 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:03.744633 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:03.744656 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:03.752135 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40523
I20260619 16:49:03.752199 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40523 every 8 connection(s)
I20260619 16:49:03.752580 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:49:03.754658 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 774
I20260619 16:49:03.754755 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781887741903642-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:49:03.755004 392 external_mini_cluster.cc:1428] Setting key fdec27cce9c61a78e5ea8d44fce1a1b3
Jun 19 16:49:03 dist-test-slave-jsw9 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:03.763962 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:03.753742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36239 (local address 127.0.98.62:45377)
0619 16:49:03.753857 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:03.753860 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:03.754067 (+ 207us) server_negotiation.cc:407] Connection header received
0619 16:49:03.755138 (+ 1071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:03.755140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:03.755156 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:03.755207 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:03.756574 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:03.757194 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:03.758478 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:03.758677 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:03.758735 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:03.761268 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:03.761270 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:03.761271 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:03.761279 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:03.762569 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:03.762995 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:03.762996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:03.762996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:03.763051 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:03.763280 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:03.763281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:03.763281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:03.763404 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:03.763462 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:03.763759 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 16:49:03.763847 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 16:49:03.764179 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:03.753670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45377 (local address 127.0.98.3:36239)
0619 16:49:03.753903 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:03.753950 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:03.754990 (+ 1040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:03.755304 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:03.755310 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:03.756141 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:03.756493 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:03.756503 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:03.757278 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:03.757279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:03.758402 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:03.758405 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:03.758745 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:03.758746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:03.758892 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:03.759555 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:03.759566 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:03.761163 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:03.762647 (+ 1484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:03.762656 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:03.762659 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:03.762911 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:03.763108 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:03.763109 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:03.763109 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:03.763226 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:03.763467 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:03.763474 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:03.763633 (+ 159us) client_negotiation.cc:769] Sending connection context
0619 16:49:03.763731 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 16:49:03.763919 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":103,"threads_started":1}
I20260619 16:49:03.764792 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45377
I20260619 16:49:03.764925 903 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:03.765151 903 heartbeater.cc:511] Master 127.0.98.62:45377 requested a full tablet report, sending...
I20260619 16:49:03.765796 444 ts_manager.cc:194] Registered new tserver with Master: d334f60a413348acb4272e508f8276d9 (127.0.98.3:40523)
I20260619 16:49:03.766264 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36239
I20260619 16:49:03.767339 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 16:49:03 dist-test-slave-jsw9 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887743, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 16:49:03 dist-test-slave-jsw9 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887743, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 16:49:03.796178 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:03.785961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54318 (local address 127.0.98.1:42399)
0619 16:49:03.786140 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:03.786146 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:03.786158 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:49:03.786210 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:03.786213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:03.786233 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:03.786322 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:03.787013 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:03.787837 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:03.788771 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:03.789167 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:03.789234 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:03.791877 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:03.791889 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:03.791889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:03.791898 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:03.793926 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:03.794964 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:03.794970 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:03.794971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:03.795049 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:03.795352 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:03.795353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:03.795353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:03.795587 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:03.795730 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:03.795972 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 16:49:03.796071 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":91,"threads_started":1}
W20260619 16:49:03.796665 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54318
Jun 19 16:49:03 dist-test-slave-jsw9 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 16:49:03 dist-test-slave-jsw9 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 16:49:03.822857 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:03.814664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54330 (local address 127.0.98.1:42399)
0619 16:49:03.814722 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:03.814724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:03.814736 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:49:03.814848 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:03.814850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:03.814867 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:03.814933 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:03.815594 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:03.816151 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:03.816988 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:03.817195 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:03.817242 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:03.820140 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:03.820142 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:03.820143 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:03.820152 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:03.821574 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:03.821958 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:03.821959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:03.821959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:03.822009 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:03.822299 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:03.822300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:03.822300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:03.822456 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:03.822576 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:03.822692 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 16:49:03.822774 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 16:49:03.823632 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 506
I20260619 16:49:03.828662 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 640
I20260619 16:49:03.833899 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 774
I20260619 16:49:03.838268 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 412
2026-06-19T16:49:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1940 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:49:03 dist-test-slave-jsw9 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:49:03 dist-test-slave-jsw9 krb5kdc[917](info): set up 2 sockets
Jun 19 16:49:03 dist-test-slave-jsw9 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260619 16:49:04.814043 392 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:49:04 dist-test-slave-jsw9 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:49:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:49:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:49:04.946148 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44595
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44595
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:05.045408 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:05.045684 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:05.045709 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:05.047111 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:05.047183 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:05.047197 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:05.047209 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:05.047220 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:49:05.049228 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:05.049532 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:05.049759 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:05.055984 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:05.056005 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:05.055944 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:05.056284 930 server_base.cc:1034] running on GCE node
I20260619 16:49:05.056602 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:05.056975 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:05.058147 930 hybrid_clock.cc:648] HybridClock initialized: now 1781887745058114 us; error 52 us; skew 500 ppm
Jun 19 16:49:05 dist-test-slave-jsw9 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:05.061232 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:49:05.061849 930 webserver.cc:460] Webserver started at http://127.0.98.62:36525/ using document root <none> and password file <none>
I20260619 16:49:05.062119 930 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:05.062173 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:05.062367 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:05.063642 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "722fe16f8ede4f63b06bed69d9fc578d"
format_stamp: "Formatted at 2026-06-19 16:49:05 on dist-test-slave-jsw9"
server_key: "f8f4fab9327268914acae2b329f3db97"
server_key_iv: "24140d43b84dd7518ce0c13352551d94"
server_key_version: "encryptionkey@0"
I20260619 16:49:05.064035 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "722fe16f8ede4f63b06bed69d9fc578d"
format_stamp: "Formatted at 2026-06-19 16:49:05 on dist-test-slave-jsw9"
server_key: "f8f4fab9327268914acae2b329f3db97"
server_key_iv: "24140d43b84dd7518ce0c13352551d94"
server_key_version: "encryptionkey@0"
I20260619 16:49:05.065621 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 16:49:05.066561 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:05.066780 930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:05.066887 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "722fe16f8ede4f63b06bed69d9fc578d"
format_stamp: "Formatted at 2026-06-19 16:49:05 on dist-test-slave-jsw9"
server_key: "f8f4fab9327268914acae2b329f3db97"
server_key_iv: "24140d43b84dd7518ce0c13352551d94"
server_key_version: "encryptionkey@0"
I20260619 16:49:05.066972 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:05.076699 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:05.079607 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:05.079761 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:05.084702 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44595
I20260619 16:49:05.084731 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44595 every 8 connection(s)
I20260619 16:49:05.085147 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:05.086216 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:05.089064 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 722fe16f8ede4f63b06bed69d9fc578d: Bootstrap starting.
I20260619 16:49:05.089787 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 722fe16f8ede4f63b06bed69d9fc578d: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:05.090118 998 log.cc:824] T 00000000000000000000000000000000 P 722fe16f8ede4f63b06bed69d9fc578d: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:05.090981 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 722fe16f8ede4f63b06bed69d9fc578d: No bootstrap required, opened a new log
I20260619 16:49:05.092567 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 930
I20260619 16:49:05.092662 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:49:05.092823 392 external_mini_cluster.cc:1428] Setting key d2ded093185842bb60e0c89903d9f1bd
I20260619 16:49:05.093057 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 722fe16f8ede4f63b06bed69d9fc578d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "722fe16f8ede4f63b06bed69d9fc578d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44595 } }
I20260619 16:49:05.093305 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 722fe16f8ede4f63b06bed69d9fc578d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:05.093350 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 722fe16f8ede4f63b06bed69d9fc578d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 722fe16f8ede4f63b06bed69d9fc578d, State: Initialized, Role: FOLLOWER
I20260619 16:49:05.093660 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 722fe16f8ede4f63b06bed69d9fc578d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "722fe16f8ede4f63b06bed69d9fc578d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44595 } }
I20260619 16:49:05.093788 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 722fe16f8ede4f63b06bed69d9fc578d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:05.093835 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 722fe16f8ede4f63b06bed69d9fc578d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:05.093887 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 722fe16f8ede4f63b06bed69d9fc578d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:05.094975 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 722fe16f8ede4f63b06bed69d9fc578d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "722fe16f8ede4f63b06bed69d9fc578d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44595 } }
I20260619 16:49:05.095127 998 leader_election.cc:302] T 00000000000000000000000000000000 P 722fe16f8ede4f63b06bed69d9fc578d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 722fe16f8ede4f63b06bed69d9fc578d; no voters:
I20260619 16:49:05.095322 998 leader_election.cc:288] T 00000000000000000000000000000000 P 722fe16f8ede4f63b06bed69d9fc578d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:49:05.095376 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 722fe16f8ede4f63b06bed69d9fc578d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:05.095549 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 722fe16f8ede4f63b06bed69d9fc578d [term 1 LEADER]: Becoming Leader. State: Replica: 722fe16f8ede4f63b06bed69d9fc578d, State: Running, Role: LEADER
I20260619 16:49:05.095640 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 722fe16f8ede4f63b06bed69d9fc578d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:05.095705 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 722fe16f8ede4f63b06bed69d9fc578d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "722fe16f8ede4f63b06bed69d9fc578d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44595 } }
I20260619 16:49:05.096124 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 722fe16f8ede4f63b06bed69d9fc578d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 722fe16f8ede4f63b06bed69d9fc578d. Latest consensus state: current_term: 1 leader_uuid: "722fe16f8ede4f63b06bed69d9fc578d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "722fe16f8ede4f63b06bed69d9fc578d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44595 } } }
I20260619 16:49:05.096203 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 722fe16f8ede4f63b06bed69d9fc578d [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:05.096364 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 722fe16f8ede4f63b06bed69d9fc578d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "722fe16f8ede4f63b06bed69d9fc578d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "722fe16f8ede4f63b06bed69d9fc578d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44595 } } }
I20260619 16:49:05.096438 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 722fe16f8ede4f63b06bed69d9fc578d [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:05.096570 1010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:05.097359 1010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:05.099413 1010 catalog_manager.cc:1347] Generated new cluster ID: 7643469282c44d7395011a62cf91df5b
I20260619 16:49:05.099478 1010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 16:49:05 dist-test-slave-jsw9 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:05.106575 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:05.093562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39848 (local address 127.0.98.62:44595)
0619 16:49:05.093818 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:05.093832 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:05.093875 (+ 43us) server_negotiation.cc:407] Connection header received
0619 16:49:05.094032 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:05.094042 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:05.094116 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:05.094327 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:05.095065 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.096673 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.097892 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.098547 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.098690 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:05.101834 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:05.101840 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:05.101841 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:05.101858 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:05.104614 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:05.105107 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:05.105116 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:05.105117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:05.105247 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:05.105469 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:05.105469 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:05.105470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:05.105816 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:05.105984 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:05.106109 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 16:49:05.106303 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":136,"threads_started":1}
I20260619 16:49:05.114743 1010 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:49:05.115342 1010 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:05.118218 1010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 722fe16f8ede4f63b06bed69d9fc578d: Generated new TSK 0
I20260619 16:49:05.118482 1010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:49:05.154377 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44595
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:05.252033 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:05.252312 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:05.252358 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:05.253947 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:05.254015 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:05.254063 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:49:05.256037 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44595
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:05.256552 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:05.256937 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:05.263302 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:05.263404 1024 server_base.cc:1034] running on GCE node
W20260619 16:49:05.263326 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:05.263310 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:05.263849 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:05.264138 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:05.265445 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781887745265422 us; error 34 us; skew 500 ppm
Jun 19 16:49:05 dist-test-slave-jsw9 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:05.268878 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:49:05.269464 1024 webserver.cc:460] Webserver started at http://127.0.98.1:41017/ using document root <none> and password file <none>
I20260619 16:49:05.269747 1024 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:05.269805 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:05.269963 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:05.271567 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-0/data/instance:
uuid: "f98ebd43d4524cf88d8371df1b4ae4d7"
format_stamp: "Formatted at 2026-06-19 16:49:05 on dist-test-slave-jsw9"
server_key: "626ef8274ca577fa71fbbf60bd34b50b"
server_key_iv: "0f48fa630f771b5668b94aa6e35936c5"
server_key_version: "encryptionkey@0"
I20260619 16:49:05.272011 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f98ebd43d4524cf88d8371df1b4ae4d7"
format_stamp: "Formatted at 2026-06-19 16:49:05 on dist-test-slave-jsw9"
server_key: "626ef8274ca577fa71fbbf60bd34b50b"
server_key_iv: "0f48fa630f771b5668b94aa6e35936c5"
server_key_version: "encryptionkey@0"
I20260619 16:49:05.274399 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260619 16:49:05.275596 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:05.275894 1024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:05.276022 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-0/wal
uuid: "f98ebd43d4524cf88d8371df1b4ae4d7"
format_stamp: "Formatted at 2026-06-19 16:49:05 on dist-test-slave-jsw9"
server_key: "626ef8274ca577fa71fbbf60bd34b50b"
server_key_iv: "0f48fa630f771b5668b94aa6e35936c5"
server_key_version: "encryptionkey@0"
I20260619 16:49:05.276218 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:05.306515 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:05.309052 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:05.309223 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:05.309501 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:05.309875 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:05.309912 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:05.309947 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:05.309964 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:05.316716 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40143
I20260619 16:49:05.316778 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40143 every 8 connection(s)
I20260619 16:49:05.317173 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:49:05.321557 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 1024
I20260619 16:49:05.321668 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:49:05.321844 392 external_mini_cluster.cc:1428] Setting key 4844d20d668f5dd05bd1954a971e9f21
Jun 19 16:49:05 dist-test-slave-jsw9 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:05.329646 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:05.318103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52665 (local address 127.0.98.62:44595)
0619 16:49:05.318229 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:05.318232 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:05.318463 (+ 231us) server_negotiation.cc:407] Connection header received
0619 16:49:05.319099 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:05.319101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:05.319122 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:05.319175 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:05.320347 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.320911 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.322146 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.322454 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.322519 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:05.325332 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:05.325334 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:05.325334 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:05.325349 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:05.327175 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:05.327844 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:05.327846 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:05.327846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:05.327919 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:05.328393 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:05.328396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:05.328396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:05.328623 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:05.328863 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:05.329447 (+ 584us) server_negotiation.cc:299] Negotiation successful
0619 16:49:05.329555 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:49:05.329927 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:05.318056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44595 (local address 127.0.98.1:52665)
0619 16:49:05.318314 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:05.318363 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:05.318983 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:05.319275 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:05.319281 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:05.319959 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:05.320218 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:05.320224 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.321019 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.321021 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:05.322053 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:05.322056 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.322536 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.322537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:05.322691 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:05.323310 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:05.323321 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:05.325178 (+ 1857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:05.327293 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:05.327307 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:05.327311 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:05.327702 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:05.328003 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:05.328004 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:05.328005 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:05.328206 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:05.328868 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:05.328877 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:05.329154 (+ 277us) client_negotiation.cc:769] Sending connection context
0619 16:49:05.329329 (+ 175us) client_negotiation.cc:240] Negotiation successful
0619 16:49:05.329578 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":129,"threads_started":1}
I20260619 16:49:05.330590 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44595
I20260619 16:49:05.330724 1153 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:05.330960 1153 heartbeater.cc:511] Master 127.0.98.62:44595 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 16:49:05.331664 962 ts_manager.cc:194] Registered new tserver with Master: f98ebd43d4524cf88d8371df1b4ae4d7 (127.0.98.1:40143)
I20260619 16:49:05.332387 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52665
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:49:05.357600 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44595
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:05.451000 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:05.451313 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:05.451360 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:05.453226 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:05.453334 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:05.453387 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:49:05.455618 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44595
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:05.456001 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:05.456310 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:05.462623 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:05.462651 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:05.462623 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:05.462841 1158 server_base.cc:1034] running on GCE node
I20260619 16:49:05.463156 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:05.463469 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:05.464684 1158 hybrid_clock.cc:648] HybridClock initialized: now 1781887745464662 us; error 44 us; skew 500 ppm
Jun 19 16:49:05 dist-test-slave-jsw9 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:05.467965 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:49:05.468550 1158 webserver.cc:460] Webserver started at http://127.0.98.2:40479/ using document root <none> and password file <none>
I20260619 16:49:05.468811 1158 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:05.468868 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:05.469012 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:05.470508 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-1/data/instance:
uuid: "95084b392ccc40d797a0a2536bfd25f2"
format_stamp: "Formatted at 2026-06-19 16:49:05 on dist-test-slave-jsw9"
server_key: "0c636b94dbd7a29e22fe835542da41d9"
server_key_iv: "f3953fc9b4f092a2f4680552407a2fed"
server_key_version: "encryptionkey@0"
I20260619 16:49:05.470959 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-1/wal/instance:
uuid: "95084b392ccc40d797a0a2536bfd25f2"
format_stamp: "Formatted at 2026-06-19 16:49:05 on dist-test-slave-jsw9"
server_key: "0c636b94dbd7a29e22fe835542da41d9"
server_key_iv: "f3953fc9b4f092a2f4680552407a2fed"
server_key_version: "encryptionkey@0"
I20260619 16:49:05.472563 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 16:49:05.473416 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:05.473577 1158 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:05.473677 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-1/wal
uuid: "95084b392ccc40d797a0a2536bfd25f2"
format_stamp: "Formatted at 2026-06-19 16:49:05 on dist-test-slave-jsw9"
server_key: "0c636b94dbd7a29e22fe835542da41d9"
server_key_iv: "f3953fc9b4f092a2f4680552407a2fed"
server_key_version: "encryptionkey@0"
I20260619 16:49:05.473754 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:05.503026 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:05.505546 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:05.505718 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:05.506017 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:05.506476 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:05.506515 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:05.506548 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:05.506565 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:05.513173 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38451
I20260619 16:49:05.513216 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38451 every 8 connection(s)
I20260619 16:49:05.513598 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:49:05.514865 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 1158
I20260619 16:49:05.514950 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:49:05.515134 392 external_mini_cluster.cc:1428] Setting key 264941bef1fd88b408d4a97f68f06bf3
Jun 19 16:49:05 dist-test-slave-jsw9 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 16:49:05.524499 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:05.514489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37533 (local address 127.0.98.62:44595)
0619 16:49:05.514586 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:05.514588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:05.514787 (+ 199us) server_negotiation.cc:407] Connection header received
0619 16:49:05.515502 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:05.515503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:05.515518 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:05.515566 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:05.516746 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.517362 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.518574 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.518867 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.518940 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:05.521349 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:05.521351 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:05.521351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:05.521362 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:05.522983 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:05.523428 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:05.523430 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:05.523430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:05.523498 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:05.523725 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:05.523726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:05.523726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:05.523899 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:05.523968 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:05.524320 (+ 352us) server_negotiation.cc:299] Negotiation successful
0619 16:49:05.524417 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 16:49:05.524631 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:05.514453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44595 (local address 127.0.98.2:37533)
0619 16:49:05.514627 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:05.514679 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:05.515396 (+ 717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:05.515659 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:05.515666 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:05.516387 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:05.516634 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:05.516640 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.517465 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.517467 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:05.518481 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:05.518485 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.518949 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.518949 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:05.519087 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:05.519677 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:05.519688 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:05.521240 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:05.523083 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:05.523091 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:05.523093 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:05.523345 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:05.523559 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:05.523559 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:05.523560 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:05.523664 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:05.523974 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:05.523981 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:05.524155 (+ 174us) client_negotiation.cc:769] Sending connection context
0619 16:49:05.524251 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 16:49:05.524406 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":90,"threads_started":1}
I20260619 16:49:05.525127 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44595
I20260619 16:49:05.525238 1287 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:05.525420 1287 heartbeater.cc:511] Master 127.0.98.62:44595 requested a full tablet report, sending...
I20260619 16:49:05.525866 962 ts_manager.cc:194] Registered new tserver with Master: 95084b392ccc40d797a0a2536bfd25f2 (127.0.98.2:38451)
I20260619 16:49:05.526342 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37533
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:49:05.545954 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44595
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:05.640462 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:05.640738 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:05.640765 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:05.642066 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:05.642114 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:05.642144 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:49:05.643805 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44595
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:05.644068 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:05.644294 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:05.650116 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:05.650116 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:05.650147 1292 server_base.cc:1034] running on GCE node
W20260619 16:49:05.650122 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:05.650651 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:05.650933 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:05.652100 1292 hybrid_clock.cc:648] HybridClock initialized: now 1781887745652085 us; error 29 us; skew 500 ppm
Jun 19 16:49:05 dist-test-slave-jsw9 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:05.655045 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:49:05.655579 1292 webserver.cc:460] Webserver started at http://127.0.98.3:38025/ using document root <none> and password file <none>
I20260619 16:49:05.655795 1292 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:05.655854 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:05.655967 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:05.657181 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-2/data/instance:
uuid: "94e430fcb93e47749160a7737a697203"
format_stamp: "Formatted at 2026-06-19 16:49:05 on dist-test-slave-jsw9"
server_key: "be98086983a00c18980313ba460840de"
server_key_iv: "92a0bbd736abd2040ef7d4d5aeae0dac"
server_key_version: "encryptionkey@0"
I20260619 16:49:05.657523 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-2/wal/instance:
uuid: "94e430fcb93e47749160a7737a697203"
format_stamp: "Formatted at 2026-06-19 16:49:05 on dist-test-slave-jsw9"
server_key: "be98086983a00c18980313ba460840de"
server_key_iv: "92a0bbd736abd2040ef7d4d5aeae0dac"
server_key_version: "encryptionkey@0"
I20260619 16:49:05.659108 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 16:49:05.660180 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:05.660501 1292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:05.660598 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-2/wal
uuid: "94e430fcb93e47749160a7737a697203"
format_stamp: "Formatted at 2026-06-19 16:49:05 on dist-test-slave-jsw9"
server_key: "be98086983a00c18980313ba460840de"
server_key_iv: "92a0bbd736abd2040ef7d4d5aeae0dac"
server_key_version: "encryptionkey@0"
I20260619 16:49:05.660677 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:05.677141 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:05.679744 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:05.679890 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:05.680119 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:05.680480 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:05.680518 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:05.680552 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:05.680576 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:05.687047 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32985
I20260619 16:49:05.687117 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32985 every 8 connection(s)
I20260619 16:49:05.687479 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:49:05.692128 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 1292
I20260619 16:49:05.692257 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781887741903642-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:49:05.692498 392 external_mini_cluster.cc:1428] Setting key 94b22243a98a2632b22939906c226af4
Jun 19 16:49:05 dist-test-slave-jsw9 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:05.698215 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:05.688357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49627 (local address 127.0.98.62:44595)
0619 16:49:05.688446 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:05.688448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:05.688624 (+ 176us) server_negotiation.cc:407] Connection header received
0619 16:49:05.689246 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:05.689247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:05.689259 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:05.689326 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:05.690457 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.691003 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.692298 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.692599 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.692687 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:05.695270 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:05.695271 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:05.695272 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:05.695280 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:05.696670 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:05.697131 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:05.697132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:05.697133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:05.697190 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:05.697458 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:05.697459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:05.697459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:05.697592 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:05.697650 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:05.697968 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 16:49:05.698080 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:49:05.698444 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:05.688284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44595 (local address 127.0.98.3:49627)
0619 16:49:05.688461 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:05.688511 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:05.689086 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:05.689396 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:05.689400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:05.690032 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:05.690332 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:05.690339 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.691124 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.691125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:05.692161 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:05.692165 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.692693 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.692695 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:05.692866 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:05.693479 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:05.693491 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:05.695182 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:05.696785 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:05.696795 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:05.696798 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:05.697056 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:05.697249 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:05.697250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:05.697251 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:05.697391 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:05.697651 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:05.697659 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:05.697832 (+ 173us) client_negotiation.cc:769] Sending connection context
0619 16:49:05.697934 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 16:49:05.698151 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
I20260619 16:49:05.699060 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44595
I20260619 16:49:05.699184 1421 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:05.699436 1421 heartbeater.cc:511] Master 127.0.98.62:44595 requested a full tablet report, sending...
I20260619 16:49:05.699904 962 ts_manager.cc:194] Registered new tserver with Master: 94e430fcb93e47749160a7737a697203 (127.0.98.3:32985)
I20260619 16:49:05.700326 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49627
I20260619 16:49:05.705116 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:49:05.713547 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:05.707130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39858 (local address 127.0.98.62:44595)
0619 16:49:05.707218 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:05.707220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:05.707280 (+ 60us) server_negotiation.cc:407] Connection header received
0619 16:49:05.707328 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:05.707329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:05.707339 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:05.707392 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:05.708092 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.708729 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.709551 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.709747 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.709809 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:05.711159 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:05.711160 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:05.711160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:05.711167 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:05.712457 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:05.712800 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:05.712801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:05.712801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:05.712859 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:05.713019 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:05.713019 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:05.713020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:05.713182 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:05.713258 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:05.713387 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 16:49:05.713462 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 16:49:05.716200 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39858:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:49:05.718166 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:49:05.725867 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:05.721419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32985 (local address 127.0.0.1:36278)
0619 16:49:05.721585 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:05.721623 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:05.721707 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:05.722099 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:05.722103 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:05.722105 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:05.722386 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:05.722391 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.723440 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.723441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:05.724550 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:05.724553 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.725517 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.725517 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:05.725603 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:05.725619 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 16:49:05.725702 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 16:49:05.725770 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":82,"threads_started":1}
I20260619 16:49:05.725874 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:05.721561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36278 (local address 127.0.98.3:32985)
0619 16:49:05.721909 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:05.721916 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:05.721927 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:49:05.721971 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:05.721973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:05.721991 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:05.722082 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:05.722486 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.723343 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.724633 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.725431 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.725502 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:05.725567 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:05.725726 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 16:49:05.725772 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":109,"threads_started":1}
I20260619 16:49:05.726047 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:05.721659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38451 (local address 127.0.0.1:57044)
0619 16:49:05.721943 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:05.721953 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:05.722003 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:05.722567 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:05.722568 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:05.722570 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:05.722623 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:05.722624 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.723990 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.723991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:05.724992 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:05.724994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.725818 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.725818 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:05.725894 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:05.725898 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:49:05.725942 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 16:49:05.725984 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":93,"thread_start_us":84,"threads_started":1}
I20260619 16:49:05.726296 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:05.721828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57044 (local address 127.0.98.2:38451)
0619 16:49:05.722370 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:05.722373 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:05.722383 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:49:05.722427 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:05.722429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:05.722446 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:05.722552 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:05.722704 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.723874 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.725073 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.725658 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.725717 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:05.725786 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:05.726128 (+ 342us) server_negotiation.cc:299] Negotiation successful
0619 16:49:05.726215 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":108,"threads_started":1}
I20260619 16:49:05.726627 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:05.722083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58578 (local address 127.0.98.1:40143)
0619 16:49:05.722267 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:05.722271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:05.722412 (+ 141us) server_negotiation.cc:407] Connection header received
0619 16:49:05.722525 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:05.722527 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:05.722542 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:05.722622 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:05.722763 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.723882 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.725170 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.725861 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.725909 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:05.725961 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:05.726481 (+ 520us) server_negotiation.cc:299] Negotiation successful
0619 16:49:05.726536 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":89,"threads_started":1}
I20260619 16:49:05.726871 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:05.721824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40143 (local address 127.0.0.1:58578)
0619 16:49:05.722291 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:05.722300 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:05.722441 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:05.722652 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:05.722653 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:05.722656 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:05.722704 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:05.722706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.723997 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.723998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:05.725066 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:05.725069 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.726255 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.726256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:05.726318 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:05.726322 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:49:05.726762 (+ 440us) client_negotiation.cc:240] Negotiation successful
0619 16:49:05.726807 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":26,"thread_start_us":66,"threads_started":1}
I20260619 16:49:05.726929 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 8c42ab3891bb47069f1c8ef3da0c21db (DEFAULT_TABLE table=test-table [id=36b5b74f25da49d385e08b70c33dd43c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:05.726929 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 8c42ab3891bb47069f1c8ef3da0c21db (DEFAULT_TABLE table=test-table [id=36b5b74f25da49d385e08b70c33dd43c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:05.727353 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c42ab3891bb47069f1c8ef3da0c21db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:05.727353 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c42ab3891bb47069f1c8ef3da0c21db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:05.730995 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 8c42ab3891bb47069f1c8ef3da0c21db (DEFAULT_TABLE table=test-table [id=36b5b74f25da49d385e08b70c33dd43c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:05.731647 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c42ab3891bb47069f1c8ef3da0c21db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:05.731735 1439 tablet_bootstrap.cc:492] T 8c42ab3891bb47069f1c8ef3da0c21db P 95084b392ccc40d797a0a2536bfd25f2: Bootstrap starting.
I20260619 16:49:05.732561 1439 tablet_bootstrap.cc:654] T 8c42ab3891bb47069f1c8ef3da0c21db P 95084b392ccc40d797a0a2536bfd25f2: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:05.733232 1439 log.cc:824] T 8c42ab3891bb47069f1c8ef3da0c21db P 95084b392ccc40d797a0a2536bfd25f2: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:05.733665 1438 tablet_bootstrap.cc:492] T 8c42ab3891bb47069f1c8ef3da0c21db P 94e430fcb93e47749160a7737a697203: Bootstrap starting.
I20260619 16:49:05.734556 1438 tablet_bootstrap.cc:654] T 8c42ab3891bb47069f1c8ef3da0c21db P 94e430fcb93e47749160a7737a697203: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:05.734650 1439 tablet_bootstrap.cc:492] T 8c42ab3891bb47069f1c8ef3da0c21db P 95084b392ccc40d797a0a2536bfd25f2: No bootstrap required, opened a new log
I20260619 16:49:05.734731 1439 ts_tablet_manager.cc:1397] T 8c42ab3891bb47069f1c8ef3da0c21db P 95084b392ccc40d797a0a2536bfd25f2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:49:05.735216 1438 log.cc:824] T 8c42ab3891bb47069f1c8ef3da0c21db P 94e430fcb93e47749160a7737a697203: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:05.736147 1438 tablet_bootstrap.cc:492] T 8c42ab3891bb47069f1c8ef3da0c21db P 94e430fcb93e47749160a7737a697203: No bootstrap required, opened a new log
I20260619 16:49:05.736222 1438 ts_tablet_manager.cc:1397] T 8c42ab3891bb47069f1c8ef3da0c21db P 94e430fcb93e47749160a7737a697203: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 16:49:05.737771 1442 tablet_bootstrap.cc:492] T 8c42ab3891bb47069f1c8ef3da0c21db P f98ebd43d4524cf88d8371df1b4ae4d7: Bootstrap starting.
I20260619 16:49:05.737886 1439 raft_consensus.cc:348] T 8c42ab3891bb47069f1c8ef3da0c21db P 95084b392ccc40d797a0a2536bfd25f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e430fcb93e47749160a7737a697203" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32985 } } peers { permanent_uuid: "95084b392ccc40d797a0a2536bfd25f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38451 } } peers { permanent_uuid: "f98ebd43d4524cf88d8371df1b4ae4d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40143 } }
I20260619 16:49:05.738173 1439 raft_consensus.cc:374] T 8c42ab3891bb47069f1c8ef3da0c21db P 95084b392ccc40d797a0a2536bfd25f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:05.738251 1439 raft_consensus.cc:729] T 8c42ab3891bb47069f1c8ef3da0c21db P 95084b392ccc40d797a0a2536bfd25f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95084b392ccc40d797a0a2536bfd25f2, State: Initialized, Role: FOLLOWER
I20260619 16:49:05.738418 1439 consensus_queue.cc:260] T 8c42ab3891bb47069f1c8ef3da0c21db P 95084b392ccc40d797a0a2536bfd25f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e430fcb93e47749160a7737a697203" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32985 } } peers { permanent_uuid: "95084b392ccc40d797a0a2536bfd25f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38451 } } peers { permanent_uuid: "f98ebd43d4524cf88d8371df1b4ae4d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40143 } }
I20260619 16:49:05.738776 1287 heartbeater.cc:503] Master 127.0.98.62:44595 was elected leader, sending a full tablet report...
I20260619 16:49:05.738794 1439 ts_tablet_manager.cc:1428] T 8c42ab3891bb47069f1c8ef3da0c21db P 95084b392ccc40d797a0a2536bfd25f2: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260619 16:49:05.739046 1442 tablet_bootstrap.cc:654] T 8c42ab3891bb47069f1c8ef3da0c21db P f98ebd43d4524cf88d8371df1b4ae4d7: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:05.739149 1438 raft_consensus.cc:348] T 8c42ab3891bb47069f1c8ef3da0c21db P 94e430fcb93e47749160a7737a697203 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e430fcb93e47749160a7737a697203" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32985 } } peers { permanent_uuid: "95084b392ccc40d797a0a2536bfd25f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38451 } } peers { permanent_uuid: "f98ebd43d4524cf88d8371df1b4ae4d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40143 } }
I20260619 16:49:05.739365 1438 raft_consensus.cc:374] T 8c42ab3891bb47069f1c8ef3da0c21db P 94e430fcb93e47749160a7737a697203 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:05.739423 1438 raft_consensus.cc:729] T 8c42ab3891bb47069f1c8ef3da0c21db P 94e430fcb93e47749160a7737a697203 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94e430fcb93e47749160a7737a697203, State: Initialized, Role: FOLLOWER
I20260619 16:49:05.739567 1438 consensus_queue.cc:260] T 8c42ab3891bb47069f1c8ef3da0c21db P 94e430fcb93e47749160a7737a697203 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e430fcb93e47749160a7737a697203" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32985 } } peers { permanent_uuid: "95084b392ccc40d797a0a2536bfd25f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38451 } } peers { permanent_uuid: "f98ebd43d4524cf88d8371df1b4ae4d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40143 } }
I20260619 16:49:05.739715 1442 log.cc:824] T 8c42ab3891bb47069f1c8ef3da0c21db P f98ebd43d4524cf88d8371df1b4ae4d7: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:05.739915 1438 ts_tablet_manager.cc:1428] T 8c42ab3891bb47069f1c8ef3da0c21db P 94e430fcb93e47749160a7737a697203: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260619 16:49:05.740069 1421 heartbeater.cc:503] Master 127.0.98.62:44595 was elected leader, sending a full tablet report...
I20260619 16:49:05.740790 1442 tablet_bootstrap.cc:492] T 8c42ab3891bb47069f1c8ef3da0c21db P f98ebd43d4524cf88d8371df1b4ae4d7: No bootstrap required, opened a new log
I20260619 16:49:05.740870 1442 ts_tablet_manager.cc:1397] T 8c42ab3891bb47069f1c8ef3da0c21db P f98ebd43d4524cf88d8371df1b4ae4d7: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 16:49:05.742664 1442 raft_consensus.cc:348] T 8c42ab3891bb47069f1c8ef3da0c21db P f98ebd43d4524cf88d8371df1b4ae4d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e430fcb93e47749160a7737a697203" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32985 } } peers { permanent_uuid: "95084b392ccc40d797a0a2536bfd25f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38451 } } peers { permanent_uuid: "f98ebd43d4524cf88d8371df1b4ae4d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40143 } }
I20260619 16:49:05.742844 1442 raft_consensus.cc:374] T 8c42ab3891bb47069f1c8ef3da0c21db P f98ebd43d4524cf88d8371df1b4ae4d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:05.742868 1442 raft_consensus.cc:729] T 8c42ab3891bb47069f1c8ef3da0c21db P f98ebd43d4524cf88d8371df1b4ae4d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f98ebd43d4524cf88d8371df1b4ae4d7, State: Initialized, Role: FOLLOWER
I20260619 16:49:05.742990 1442 consensus_queue.cc:260] T 8c42ab3891bb47069f1c8ef3da0c21db P f98ebd43d4524cf88d8371df1b4ae4d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e430fcb93e47749160a7737a697203" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32985 } } peers { permanent_uuid: "95084b392ccc40d797a0a2536bfd25f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38451 } } peers { permanent_uuid: "f98ebd43d4524cf88d8371df1b4ae4d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40143 } }
I20260619 16:49:05.743429 1442 ts_tablet_manager.cc:1428] T 8c42ab3891bb47069f1c8ef3da0c21db P f98ebd43d4524cf88d8371df1b4ae4d7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 16:49:05.743477 1153 heartbeater.cc:503] Master 127.0.98.62:44595 was elected leader, sending a full tablet report...
I20260619 16:49:05.758863 1443 raft_consensus.cc:482] T 8c42ab3891bb47069f1c8ef3da0c21db P 95084b392ccc40d797a0a2536bfd25f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:05.759044 1443 raft_consensus.cc:504] T 8c42ab3891bb47069f1c8ef3da0c21db P 95084b392ccc40d797a0a2536bfd25f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e430fcb93e47749160a7737a697203" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32985 } } peers { permanent_uuid: "95084b392ccc40d797a0a2536bfd25f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38451 } } peers { permanent_uuid: "f98ebd43d4524cf88d8371df1b4ae4d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40143 } }
I20260619 16:49:05.759513 1443 leader_election.cc:288] T 8c42ab3891bb47069f1c8ef3da0c21db P 95084b392ccc40d797a0a2536bfd25f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94e430fcb93e47749160a7737a697203 (127.0.98.3:32985), f98ebd43d4524cf88d8371df1b4ae4d7 (127.0.98.1:40143)
I20260619 16:49:05.763825 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:05.759708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32985 (local address 127.0.98.2:44861)
0619 16:49:05.759762 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:05.759776 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:05.759828 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:05.760019 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:05.760020 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:05.760023 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:05.760114 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:05.760117 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.761101 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.761103 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:05.762524 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:05.762529 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.763495 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.763495 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:05.763573 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:05.763575 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 16:49:05.763638 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 16:49:05.763694 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":54}
I20260619 16:49:05.763955 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:05.759831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44861 (local address 127.0.98.3:32985)
0619 16:49:05.759884 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:05.759886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:05.759896 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:49:05.759929 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:05.759930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:05.759944 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:05.760001 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:05.760193 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.760984 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.762651 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.763356 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.763430 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:05.763444 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:05.763675 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 16:49:05.763849 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 16:49:05.763830 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:05.759819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40143 (local address 127.0.98.2:56009)
0619 16:49:05.760082 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:05.760093 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:05.760145 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:05.760341 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:05.760341 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:05.760343 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:05.760399 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:05.760400 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.761236 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.761237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:05.762525 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:05.762529 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.763476 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.763478 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:05.763569 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:05.763575 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 16:49:05.763672 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 16:49:05.763706 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":4,"thread_start_us":170,"threads_started":1}
I20260619 16:49:05.763984 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:05.760075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56009 (local address 127.0.98.1:40143)
0619 16:49:05.760137 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:05.760139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:05.760148 (+ 9us) server_negotiation.cc:407] Connection header received
0619 16:49:05.760216 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:05.760217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:05.760231 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:05.760337 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:05.760476 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.761144 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.762652 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.763344 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.763430 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:05.763444 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:05.763681 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 16:49:05.763903 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 16:49:05.764209 1288 tablet.cc:2369] T 8c42ab3891bb47069f1c8ef3da0c21db P 95084b392ccc40d797a0a2536bfd25f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:49:05.764386 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c42ab3891bb47069f1c8ef3da0c21db" candidate_uuid: "95084b392ccc40d797a0a2536bfd25f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f98ebd43d4524cf88d8371df1b4ae4d7" is_pre_election: true
I20260619 16:49:05.764603 1107 raft_consensus.cc:2454] T 8c42ab3891bb47069f1c8ef3da0c21db P f98ebd43d4524cf88d8371df1b4ae4d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95084b392ccc40d797a0a2536bfd25f2 in term 0.
I20260619 16:49:05.764708 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c42ab3891bb47069f1c8ef3da0c21db" candidate_uuid: "95084b392ccc40d797a0a2536bfd25f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94e430fcb93e47749160a7737a697203" is_pre_election: true
I20260619 16:49:05.764879 1375 raft_consensus.cc:2454] T 8c42ab3891bb47069f1c8ef3da0c21db P 94e430fcb93e47749160a7737a697203 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95084b392ccc40d797a0a2536bfd25f2 in term 0.
I20260619 16:49:05.764894 1176 leader_election.cc:302] T 8c42ab3891bb47069f1c8ef3da0c21db P 95084b392ccc40d797a0a2536bfd25f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95084b392ccc40d797a0a2536bfd25f2, f98ebd43d4524cf88d8371df1b4ae4d7; no voters:
I20260619 16:49:05.765064 1443 raft_consensus.cc:2781] T 8c42ab3891bb47069f1c8ef3da0c21db P 95084b392ccc40d797a0a2536bfd25f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:49:05.765125 1443 raft_consensus.cc:482] T 8c42ab3891bb47069f1c8ef3da0c21db P 95084b392ccc40d797a0a2536bfd25f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:49:05.765148 1443 raft_consensus.cc:3037] T 8c42ab3891bb47069f1c8ef3da0c21db P 95084b392ccc40d797a0a2536bfd25f2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:05.766027 1443 raft_consensus.cc:504] T 8c42ab3891bb47069f1c8ef3da0c21db P 95084b392ccc40d797a0a2536bfd25f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e430fcb93e47749160a7737a697203" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32985 } } peers { permanent_uuid: "95084b392ccc40d797a0a2536bfd25f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38451 } } peers { permanent_uuid: "f98ebd43d4524cf88d8371df1b4ae4d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40143 } }
I20260619 16:49:05.766170 1443 leader_election.cc:288] T 8c42ab3891bb47069f1c8ef3da0c21db P 95084b392ccc40d797a0a2536bfd25f2 [CANDIDATE]: Term 1 election: Requested vote from peers 94e430fcb93e47749160a7737a697203 (127.0.98.3:32985), f98ebd43d4524cf88d8371df1b4ae4d7 (127.0.98.1:40143)
I20260619 16:49:05.766369 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c42ab3891bb47069f1c8ef3da0c21db" candidate_uuid: "95084b392ccc40d797a0a2536bfd25f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94e430fcb93e47749160a7737a697203"
I20260619 16:49:05.766423 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c42ab3891bb47069f1c8ef3da0c21db" candidate_uuid: "95084b392ccc40d797a0a2536bfd25f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f98ebd43d4524cf88d8371df1b4ae4d7"
I20260619 16:49:05.766505 1107 raft_consensus.cc:3037] T 8c42ab3891bb47069f1c8ef3da0c21db P f98ebd43d4524cf88d8371df1b4ae4d7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:05.766603 1375 raft_consensus.cc:3037] T 8c42ab3891bb47069f1c8ef3da0c21db P 94e430fcb93e47749160a7737a697203 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:05.767405 1107 raft_consensus.cc:2454] T 8c42ab3891bb47069f1c8ef3da0c21db P f98ebd43d4524cf88d8371df1b4ae4d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95084b392ccc40d797a0a2536bfd25f2 in term 1.
I20260619 16:49:05.767620 1176 leader_election.cc:302] T 8c42ab3891bb47069f1c8ef3da0c21db P 95084b392ccc40d797a0a2536bfd25f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95084b392ccc40d797a0a2536bfd25f2, f98ebd43d4524cf88d8371df1b4ae4d7; no voters:
I20260619 16:49:05.767719 1443 raft_consensus.cc:2781] T 8c42ab3891bb47069f1c8ef3da0c21db P 95084b392ccc40d797a0a2536bfd25f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:05.767763 1375 raft_consensus.cc:2454] T 8c42ab3891bb47069f1c8ef3da0c21db P 94e430fcb93e47749160a7737a697203 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95084b392ccc40d797a0a2536bfd25f2 in term 1.
I20260619 16:49:05.767923 1443 raft_consensus.cc:686] T 8c42ab3891bb47069f1c8ef3da0c21db P 95084b392ccc40d797a0a2536bfd25f2 [term 1 LEADER]: Becoming Leader. State: Replica: 95084b392ccc40d797a0a2536bfd25f2, State: Running, Role: LEADER
I20260619 16:49:05.768042 1443 consensus_queue.cc:237] T 8c42ab3891bb47069f1c8ef3da0c21db P 95084b392ccc40d797a0a2536bfd25f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e430fcb93e47749160a7737a697203" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32985 } } peers { permanent_uuid: "95084b392ccc40d797a0a2536bfd25f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38451 } } peers { permanent_uuid: "f98ebd43d4524cf88d8371df1b4ae4d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40143 } }
I20260619 16:49:05.769021 961 catalog_manager.cc:5495] T 8c42ab3891bb47069f1c8ef3da0c21db P 95084b392ccc40d797a0a2536bfd25f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 95084b392ccc40d797a0a2536bfd25f2 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "95084b392ccc40d797a0a2536bfd25f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94e430fcb93e47749160a7737a697203" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95084b392ccc40d797a0a2536bfd25f2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f98ebd43d4524cf88d8371df1b4ae4d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40143 } health_report { overall_health: UNKNOWN } } }
Jun 19 16:49:05 dist-test-slave-jsw9 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 16:49:05.790100 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:05.782181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58582 (local address 127.0.98.1:40143)
0619 16:49:05.782288 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:05.782290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:05.782394 (+ 104us) server_negotiation.cc:407] Connection header received
0619 16:49:05.782480 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:05.782481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:05.782496 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:05.782541 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:05.783174 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.783687 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.784444 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.784644 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.784704 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:05.787217 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:05.787231 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:05.787231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:05.787240 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:05.788918 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:05.789280 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:05.789284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:05.789284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:05.789343 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:05.789528 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:05.789529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:05.789530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:05.789697 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:05.789851 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:05.789892 (+ 41us) server_negotiation.cc:299] Negotiation successful
0619 16:49:05.789989 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
Jun 19 16:49:05 dist-test-slave-jsw9 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887745, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 16:49:05 dist-test-slave-jsw9 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887745, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 16:49:05.816963 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:05.808874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58598 (local address 127.0.98.1:40143)
0619 16:49:05.809007 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:05.809011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:05.809101 (+ 90us) server_negotiation.cc:407] Connection header received
0619 16:49:05.809187 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:05.809189 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:05.809201 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:05.809256 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:05.809917 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.810491 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.811278 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.811505 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.811567 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:05.814230 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:05.814232 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:05.814233 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:05.814243 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:05.815689 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:05.816081 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:05.816082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:05.816083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:05.816141 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:05.816319 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:05.816320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:05.816321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:05.816445 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:05.816544 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:05.816753 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 16:49:05.816849 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
W20260619 16:49:05.818070 1154 tablet.cc:2369] T 8c42ab3891bb47069f1c8ef3da0c21db P f98ebd43d4524cf88d8371df1b4ae4d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 16:49:05 dist-test-slave-jsw9 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 16:49:05 dist-test-slave-jsw9 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 16:49:05.843258 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:05.835454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58608 (local address 127.0.98.1:40143)
0619 16:49:05.835575 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:05.835578 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:05.835588 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:49:05.835620 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:05.835621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:05.835632 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:05.835684 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:05.836321 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.836843 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.837604 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:05.837805 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:05.837863 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:05.840378 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:05.840380 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:05.840380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:05.840388 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:05.841808 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:05.842343 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:05.842344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:05.842345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:05.842416 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:05.842677 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:05.842678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:05.842679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:05.842798 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:05.842884 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:05.843069 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 16:49:05.843153 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
W20260619 16:49:05.843978 1067 tablet_service.cc:2908] Rejecting scan request for tablet 8c42ab3891bb47069f1c8ef3da0c21db: Uninitialized: clean time has not yet been initialized
I20260619 16:49:05.845324 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 1024
I20260619 16:49:05.851451 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 1158
I20260619 16:49:05.856586 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 1292
I20260619 16:49:05.861034 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 930
2026-06-19T16:49:05Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2023 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:49:05 dist-test-slave-jsw9 krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:49:05 dist-test-slave-jsw9 krb5kdc[1462](info): set up 2 sockets
Jun 19 16:49:05 dist-test-slave-jsw9 krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260619 16:49:06.858605 392 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260619 16:49:07.087159 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.143s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260619 16:49:07.376544 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.289s user 0.001s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:49:07 dist-test-slave-jsw9 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:49:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:49:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:49:07.450453 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38001
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:07.550904 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:07.551209 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:07.551247 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:07.552757 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:07.552824 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 16:49:07.552842 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:07.552860 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:07.552875 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:07.552891 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:49:07.554915 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38001
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:07.555250 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:07.555521 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:07.561223 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:07.561240 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:07.561221 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:07.561902 1475 server_base.cc:1034] running on GCE node
I20260619 16:49:07.562410 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:07.562793 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:07.564002 1475 hybrid_clock.cc:648] HybridClock initialized: now 1781887747563981 us; error 40 us; skew 500 ppm
Jun 19 16:49:07 dist-test-slave-jsw9 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:07.572803 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:49:07.573644 1475 webserver.cc:460] Webserver started at http://127.0.98.62:42739/ using document root <none> and password file <none>
I20260619 16:49:07.573957 1475 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:07.574030 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:07.574224 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:07.575582 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "7ff670f5ffd941f086bf2f9c2d9955d1"
format_stamp: "Formatted at 2026-06-19 16:49:07 on dist-test-slave-jsw9"
server_key: "b226eba3414e365734437ed7c3f629ed"
server_key_iv: "72f6b3de020da23eed227f60f7379e63"
server_key_version: "encryptionkey@0"
I20260619 16:49:07.576001 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "7ff670f5ffd941f086bf2f9c2d9955d1"
format_stamp: "Formatted at 2026-06-19 16:49:07 on dist-test-slave-jsw9"
server_key: "b226eba3414e365734437ed7c3f629ed"
server_key_iv: "72f6b3de020da23eed227f60f7379e63"
server_key_version: "encryptionkey@0"
I20260619 16:49:07.577700 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 16:49:07.578701 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:07.578886 1475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 16:49:07.578989 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "7ff670f5ffd941f086bf2f9c2d9955d1"
format_stamp: "Formatted at 2026-06-19 16:49:07 on dist-test-slave-jsw9"
server_key: "b226eba3414e365734437ed7c3f629ed"
server_key_iv: "72f6b3de020da23eed227f60f7379e63"
server_key_version: "encryptionkey@0"
I20260619 16:49:07.579074 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:07.588935 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:07.591253 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:07.591441 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:07.597147 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38001
I20260619 16:49:07.597262 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38001 every 8 connection(s)
I20260619 16:49:07.597640 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:07.598542 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:07.601653 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ff670f5ffd941f086bf2f9c2d9955d1: Bootstrap starting.
I20260619 16:49:07.602783 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ff670f5ffd941f086bf2f9c2d9955d1: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:07.603377 1553 log.cc:824] T 00000000000000000000000000000000 P 7ff670f5ffd941f086bf2f9c2d9955d1: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:07.604483 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ff670f5ffd941f086bf2f9c2d9955d1: No bootstrap required, opened a new log
I20260619 16:49:07.607074 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ff670f5ffd941f086bf2f9c2d9955d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff670f5ffd941f086bf2f9c2d9955d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38001 } }
I20260619 16:49:07.607290 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ff670f5ffd941f086bf2f9c2d9955d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:07.607316 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ff670f5ffd941f086bf2f9c2d9955d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ff670f5ffd941f086bf2f9c2d9955d1, State: Initialized, Role: FOLLOWER
I20260619 16:49:07.607460 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ff670f5ffd941f086bf2f9c2d9955d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff670f5ffd941f086bf2f9c2d9955d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38001 } }
I20260619 16:49:07.607527 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ff670f5ffd941f086bf2f9c2d9955d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:07.607560 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ff670f5ffd941f086bf2f9c2d9955d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:07.607607 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ff670f5ffd941f086bf2f9c2d9955d1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:07.607847 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 1475
I20260619 16:49:07.607951 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:49:07.608120 392 external_mini_cluster.cc:1428] Setting key 980cc1896b641c7d1e6954fde9dc03c7
I20260619 16:49:07.608424 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ff670f5ffd941f086bf2f9c2d9955d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff670f5ffd941f086bf2f9c2d9955d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38001 } }
I20260619 16:49:07.608564 1553 leader_election.cc:302] T 00000000000000000000000000000000 P 7ff670f5ffd941f086bf2f9c2d9955d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ff670f5ffd941f086bf2f9c2d9955d1; no voters:
I20260619 16:49:07.608819 1553 leader_election.cc:288] T 00000000000000000000000000000000 P 7ff670f5ffd941f086bf2f9c2d9955d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:49:07.608853 1557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ff670f5ffd941f086bf2f9c2d9955d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:07.608999 1557 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ff670f5ffd941f086bf2f9c2d9955d1 [term 1 LEADER]: Becoming Leader. State: Replica: 7ff670f5ffd941f086bf2f9c2d9955d1, State: Running, Role: LEADER
I20260619 16:49:07.609109 1557 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ff670f5ffd941f086bf2f9c2d9955d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff670f5ffd941f086bf2f9c2d9955d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38001 } }
I20260619 16:49:07.609134 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ff670f5ffd941f086bf2f9c2d9955d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:07.609905 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ff670f5ffd941f086bf2f9c2d9955d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ff670f5ffd941f086bf2f9c2d9955d1. Latest consensus state: current_term: 1 leader_uuid: "7ff670f5ffd941f086bf2f9c2d9955d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff670f5ffd941f086bf2f9c2d9955d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38001 } } }
I20260619 16:49:07.610011 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ff670f5ffd941f086bf2f9c2d9955d1 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:07.610488 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ff670f5ffd941f086bf2f9c2d9955d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ff670f5ffd941f086bf2f9c2d9955d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff670f5ffd941f086bf2f9c2d9955d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38001 } } }
I20260619 16:49:07.610570 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ff670f5ffd941f086bf2f9c2d9955d1 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:07.610791 1567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:07.611536 1567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:07.613749 1567 catalog_manager.cc:1347] Generated new cluster ID: 43fa6305685b489bb66891cc782d7752
I20260619 16:49:07.613847 1567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 16:49:07 dist-test-slave-jsw9 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:07.620476 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:07.608761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50988 (local address 127.0.98.62:38001)
0619 16:49:07.609048 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:07.609063 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:07.609108 (+ 45us) server_negotiation.cc:407] Connection header received
0619 16:49:07.609268 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:07.609277 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:07.609348 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:07.609530 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:07.610784 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.612213 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.613298 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.613858 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.613974 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:07.616634 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:07.616639 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:07.616639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:07.616654 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:07.618656 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:07.619104 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:07.619108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:07.619108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:07.619236 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:07.619448 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:07.619448 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:07.619449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:07.619756 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:07.619932 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:07.620041 (+ 109us) server_negotiation.cc:299] Negotiation successful
0619 16:49:07.620206 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":73,"threads_started":1}
I20260619 16:49:07.636227 1567 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:49:07.637125 1567 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:07.647256 1567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ff670f5ffd941f086bf2f9c2d9955d1: Generated new TSK 0
I20260619 16:49:07.647545 1567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:49:07.686555 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38001
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:07.781225 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:07.781567 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:07.781610 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:07.783310 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:07.783401 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:07.783556 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:49:07.783856 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:49:07.785981 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:07.786405 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:07.786746 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:07.794786 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:07.794749 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:07.794736 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:07.795054 1579 server_base.cc:1034] running on GCE node
I20260619 16:49:07.795359 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:07.795671 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:07.796852 1579 hybrid_clock.cc:648] HybridClock initialized: now 1781887747796711 us; error 162 us; skew 500 ppm
Jun 19 16:49:07 dist-test-slave-jsw9 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:07.800025 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:49:07.800613 1579 webserver.cc:460] Webserver started at http://127.0.98.1:36597/ using document root <none> and password file <none>
I20260619 16:49:07.800860 1579 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:07.800912 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:07.801051 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:07.802408 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-0/data/instance:
uuid: "8f265f16f78f459fa0bf39bec82cee7d"
format_stamp: "Formatted at 2026-06-19 16:49:07 on dist-test-slave-jsw9"
server_key: "87c2bb120d5150dd6ed3dd84b2aeb337"
server_key_iv: "7c250973aa0964d7cba135e03e88f522"
server_key_version: "encryptionkey@0"
I20260619 16:49:07.802843 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8f265f16f78f459fa0bf39bec82cee7d"
format_stamp: "Formatted at 2026-06-19 16:49:07 on dist-test-slave-jsw9"
server_key: "87c2bb120d5150dd6ed3dd84b2aeb337"
server_key_iv: "7c250973aa0964d7cba135e03e88f522"
server_key_version: "encryptionkey@0"
I20260619 16:49:07.804499 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 16:49:07.805539 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:07.805806 1579 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:07.805923 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-0/wal
uuid: "8f265f16f78f459fa0bf39bec82cee7d"
format_stamp: "Formatted at 2026-06-19 16:49:07 on dist-test-slave-jsw9"
server_key: "87c2bb120d5150dd6ed3dd84b2aeb337"
server_key_iv: "7c250973aa0964d7cba135e03e88f522"
server_key_version: "encryptionkey@0"
I20260619 16:49:07.806005 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:07.819779 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:07.822377 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:07.822554 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:07.823290 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:07.823336 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:07.823372 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:07.823387 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:49:07 dist-test-slave-jsw9 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:07.831856 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34185
I20260619 16:49:07.832028 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34185 every 8 connection(s)
I20260619 16:49:07.832320 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:49:07.833267 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 1579
I20260619 16:49:07.833360 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:49:07.833556 392 external_mini_cluster.cc:1428] Setting key ade89138277b7af744f9f7ae9884991d
I20260619 16:49:07.835399 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:07.823735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44921 (local address 127.0.98.62:38001)
0619 16:49:07.823811 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:07.823814 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:07.824071 (+ 257us) server_negotiation.cc:407] Connection header received
0619 16:49:07.824933 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:07.824935 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:07.824957 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:07.825040 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:07.826290 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.827065 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.828932 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.829178 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.829263 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:07.832215 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:07.832217 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:07.832217 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:07.832231 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:07.833806 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:07.834285 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:07.834286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:07.834287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:07.834363 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:07.834622 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:07.834623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:07.834624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:07.834770 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:07.834877 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:07.835200 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 16:49:07.835304 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 16:49:07.835616 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:07.823715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38001 (local address 127.0.98.1:44921)
0619 16:49:07.823895 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:07.823951 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:07.824779 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:07.825151 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:07.825157 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:07.825596 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:07.826014 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:07.826024 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.827314 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.827317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:07.828828 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:07.828832 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.829289 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.829290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:07.829487 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:07.830161 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:07.830174 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:07.832081 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:07.833876 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:07.833886 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:07.833888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:07.834166 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:07.834430 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:07.834431 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:07.834431 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:07.834551 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:07.834893 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:07.834900 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:07.835049 (+ 149us) client_negotiation.cc:769] Sending connection context
0619 16:49:07.835166 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 16:49:07.835325 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":91,"threads_started":1}
I20260619 16:49:07.837195 1710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38001
I20260619 16:49:07.837352 1710 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:07.837577 1710 heartbeater.cc:511] Master 127.0.98.62:38001 requested a full tablet report, sending...
I20260619 16:49:07.838268 1506 ts_manager.cc:194] Registered new tserver with Master: 8f265f16f78f459fa0bf39bec82cee7d (127.0.98.1:34185)
I20260619 16:49:07.839052 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44921
I20260619 16:49:07.844611 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:07.837374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34127 (local address 127.0.98.62:38001)
0619 16:49:07.837422 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:07.837424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:07.838368 (+ 944us) server_negotiation.cc:407] Connection header received
0619 16:49:07.838475 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:07.838476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:07.838490 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:07.838555 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:07.839092 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.839727 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.840498 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.840701 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.840758 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:07.841836 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:07.841838 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:07.841839 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:07.841846 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:07.843263 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:07.843712 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:07.843713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:07.843714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:07.843774 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:07.844061 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:07.844062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:07.844062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:07.844194 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:07.844255 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:07.844403 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 16:49:07.844500 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 16:49:07.844640 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:07.837279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38001 (local address 127.0.98.1:34127)
0619 16:49:07.838294 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:07.838308 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:07.838398 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:07.838567 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:07.838568 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:07.838910 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:07.838997 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:07.839001 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.839833 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.839834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:07.840404 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:07.840406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:07.840766 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:07.840767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:07.840833 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:07.841390 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:07.841395 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:07.841739 (+ 344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:07.843346 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:07.843347 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:07.843347 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:07.843619 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:07.843907 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:07.843909 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:07.843909 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:07.843997 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:07.844258 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:07.844259 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:07.844309 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 16:49:07.844373 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 16:49:07.844531 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":983,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:49:07.867063 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38001
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:07.967391 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:07.967701 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:07.967733 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:07.969218 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:07.969285 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:07.969316 1719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:49:07.969336 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:49:07.971262 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:07.971560 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:07.971814 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:07.978128 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:07.978508 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:07.978645 1719 server_base.cc:1034] running on GCE node
W20260619 16:49:07.978664 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:07.979106 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:07.979450 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:07.980682 1719 hybrid_clock.cc:648] HybridClock initialized: now 1781887747980620 us; error 69 us; skew 500 ppm
Jun 19 16:49:07 dist-test-slave-jsw9 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:07.984897 1719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:49:07.985491 1719 webserver.cc:460] Webserver started at http://127.0.98.2:40317/ using document root <none> and password file <none>
I20260619 16:49:07.985733 1719 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:07.985790 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:07.985939 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:07.987254 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-1/data/instance:
uuid: "6371d7f13b834b088a3e199b2ad9c1da"
format_stamp: "Formatted at 2026-06-19 16:49:07 on dist-test-slave-jsw9"
server_key: "1b062823b8758922148ccd34ba7cf0bf"
server_key_iv: "923d4ba20f8acd826829fa7e02dd9ca6"
server_key_version: "encryptionkey@0"
I20260619 16:49:07.987684 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6371d7f13b834b088a3e199b2ad9c1da"
format_stamp: "Formatted at 2026-06-19 16:49:07 on dist-test-slave-jsw9"
server_key: "1b062823b8758922148ccd34ba7cf0bf"
server_key_iv: "923d4ba20f8acd826829fa7e02dd9ca6"
server_key_version: "encryptionkey@0"
I20260619 16:49:07.989228 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 16:49:07.990080 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:07.990290 1719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:07.990384 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-1/wal
uuid: "6371d7f13b834b088a3e199b2ad9c1da"
format_stamp: "Formatted at 2026-06-19 16:49:07 on dist-test-slave-jsw9"
server_key: "1b062823b8758922148ccd34ba7cf0bf"
server_key_iv: "923d4ba20f8acd826829fa7e02dd9ca6"
server_key_version: "encryptionkey@0"
I20260619 16:49:07.990458 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:08.006722 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:08.010959 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:08.011116 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:08.011722 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:08.011771 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:08.011807 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:08.011823 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:49:08 dist-test-slave-jsw9 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:08.019865 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36867
I20260619 16:49:08.019992 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36867 every 8 connection(s)
I20260619 16:49:08.020347 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:49:08.022924 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.012455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53689 (local address 127.0.98.62:38001)
0619 16:49:08.012521 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:08.012524 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:08.012705 (+ 181us) server_negotiation.cc:407] Connection header received
0619 16:49:08.013373 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:08.013376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:08.013393 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:08.013480 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:08.014548 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.015126 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.016454 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.016687 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.016753 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:08.019735 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:08.019746 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:08.019746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:08.019758 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:08.021406 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:08.021844 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:08.021846 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:08.021846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:08.021914 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:08.022157 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:08.022158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:08.022158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:08.022338 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:08.022401 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:08.022709 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 16:49:08.022820 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 16:49:08.023120 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.012329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38001 (local address 127.0.98.2:53689)
0619 16:49:08.012555 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:08.012601 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:08.013210 (+ 609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:08.013676 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:08.013682 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:08.014077 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:08.014446 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:08.014453 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.015236 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.015237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:08.016298 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:08.016302 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.016793 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.016794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:08.016956 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:08.017626 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:08.017638 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:08.019617 (+ 1979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:08.021502 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:08.021510 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:08.021512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:08.021765 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:08.021976 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:08.021976 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:08.021977 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:08.022096 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:08.022402 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:08.022409 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:08.022571 (+ 162us) client_negotiation.cc:769] Sending connection context
0619 16:49:08.022674 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 16:49:08.022871 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":111,"threads_started":1}
I20260619 16:49:08.023705 1850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38001
I20260619 16:49:08.023804 1850 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:08.023813 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 1719
I20260619 16:49:08.023917 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:49:08.024034 1850 heartbeater.cc:511] Master 127.0.98.62:38001 requested a full tablet report, sending...
I20260619 16:49:08.024120 392 external_mini_cluster.cc:1428] Setting key 312c0209925fa3083ea6e71e9056da95
I20260619 16:49:08.024473 1506 ts_manager.cc:194] Registered new tserver with Master: 6371d7f13b834b088a3e199b2ad9c1da (127.0.98.2:36867)
I20260619 16:49:08.024971 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53689
I20260619 16:49:08.034307 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.026285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57743 (local address 127.0.98.62:38001)
0619 16:49:08.026350 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:08.026352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:08.026361 (+ 9us) server_negotiation.cc:407] Connection header received
0619 16:49:08.026381 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:08.026382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:08.026395 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:08.026472 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:08.026994 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.027772 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.028615 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.028879 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.028971 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:08.030515 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:08.030520 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:08.030520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:08.030533 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:08.032663 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:08.033126 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:08.033128 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:08.033128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:08.033200 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:08.033457 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:08.033458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:08.033459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:08.033768 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:08.033853 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:08.034050 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 16:49:08.034154 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 16:49:08.034297 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.025266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38001 (local address 127.0.98.2:57743)
0619 16:49:08.025425 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:08.025438 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:08.025508 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:08.026496 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:08.026498 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:08.026737 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:08.026811 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:08.026814 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.027910 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.027911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:08.028520 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:08.028523 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.028989 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.028991 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:08.029066 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:08.029873 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:08.029878 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:08.030339 (+ 461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:08.032801 (+ 2462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:08.032804 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:08.032804 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:08.033028 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:08.033303 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:08.033304 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:08.033305 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:08.033383 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:08.033870 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:08.033871 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:08.033931 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 16:49:08.034001 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 16:49:08.034128 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":100,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:49:08.059763 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38001
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:08.163479 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:08.163841 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:08.163874 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:08.165530 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:08.165614 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:08.165644 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:49:08.165665 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:49:08.167721 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:08.168048 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:08.168324 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:08.174180 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:08.174180 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:08.174172 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:08.174523 1859 server_base.cc:1034] running on GCE node
I20260619 16:49:08.174758 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:08.175082 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:08.176318 1859 hybrid_clock.cc:648] HybridClock initialized: now 1781887748176214 us; error 125 us; skew 500 ppm
Jun 19 16:49:08 dist-test-slave-jsw9 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:08.179780 1859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:49:08.180399 1859 webserver.cc:460] Webserver started at http://127.0.98.3:35233/ using document root <none> and password file <none>
I20260619 16:49:08.180665 1859 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:08.180727 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:08.180872 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:08.182245 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-2/data/instance:
uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b"
format_stamp: "Formatted at 2026-06-19 16:49:08 on dist-test-slave-jsw9"
server_key: "05783794159486027b3923dfae784cb2"
server_key_iv: "622d9b825dfda7bd510d0060a5b98102"
server_key_version: "encryptionkey@0"
I20260619 16:49:08.182698 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b"
format_stamp: "Formatted at 2026-06-19 16:49:08 on dist-test-slave-jsw9"
server_key: "05783794159486027b3923dfae784cb2"
server_key_iv: "622d9b825dfda7bd510d0060a5b98102"
server_key_version: "encryptionkey@0"
I20260619 16:49:08.184312 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 16:49:08.185276 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:08.185508 1859 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:08.185606 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-2/wal
uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b"
format_stamp: "Formatted at 2026-06-19 16:49:08 on dist-test-slave-jsw9"
server_key: "05783794159486027b3923dfae784cb2"
server_key_iv: "622d9b825dfda7bd510d0060a5b98102"
server_key_version: "encryptionkey@0"
I20260619 16:49:08.185681 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:08.193414 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:08.196398 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:08.196627 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:08.197283 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:08.197328 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:08.197362 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:08.197389 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:49:08 dist-test-slave-jsw9 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:08.206058 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37749
I20260619 16:49:08.206153 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37749 every 8 connection(s)
I20260619 16:49:08.206622 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:49:08.210283 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.197814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47551 (local address 127.0.98.62:38001)
0619 16:49:08.197889 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:08.197892 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:08.198118 (+ 226us) server_negotiation.cc:407] Connection header received
0619 16:49:08.198798 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:08.198801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:08.198821 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:08.198894 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:08.199678 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.200514 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.201799 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.202118 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.202242 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:08.205933 (+ 3691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:08.205936 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:08.205937 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:08.205948 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:08.207876 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:08.208688 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:08.208689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:08.208690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:08.208774 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:08.209197 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:08.209199 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:08.209199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:08.209371 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:08.209481 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:08.209996 (+ 515us) server_negotiation.cc:299] Negotiation successful
0619 16:49:08.210119 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 16:49:08.210480 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.197770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38001 (local address 127.0.98.3:47551)
0619 16:49:08.197937 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:08.198008 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:08.198657 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:08.198967 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:08.198971 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:08.199326 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:08.199577 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:08.199584 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.200636 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.200637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:08.201709 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:08.201713 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.202254 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.202255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:08.202435 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:08.203150 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:08.203164 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:08.205779 (+ 2615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:08.208030 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:08.208039 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:08.208041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:08.208355 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:08.208903 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:08.208905 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:08.208905 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:08.209114 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:08.209464 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:08.209473 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:08.209655 (+ 182us) client_negotiation.cc:769] Sending connection context
0619 16:49:08.209937 (+ 282us) client_negotiation.cc:240] Negotiation successful
0619 16:49:08.210174 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260619 16:49:08.211231 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38001
I20260619 16:49:08.211364 1990 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:08.211791 1990 heartbeater.cc:511] Master 127.0.98.62:38001 requested a full tablet report, sending...
I20260619 16:49:08.212702 1506 ts_manager.cc:194] Registered new tserver with Master: 4ee5b1c2133f44d0b27c33bebfb6ad4b (127.0.98.3:37749)
I20260619 16:49:08.213168 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47551
I20260619 16:49:08.216679 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 1859
I20260619 16:49:08.216823 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:49:08.217000 392 external_mini_cluster.cc:1428] Setting key 2f521dbe3fbeac28511309f584526698
I20260619 16:49:08.218567 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:49:08.220537 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.213100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47573 (local address 127.0.98.62:38001)
0619 16:49:08.213438 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:08.213443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:08.213467 (+ 24us) server_negotiation.cc:407] Connection header received
0619 16:49:08.213509 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:08.213511 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:08.213532 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:08.213644 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:08.214243 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.214774 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.215622 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.215855 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.215928 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:08.217234 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:08.217292 (+ 58us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:08.217293 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:08.217308 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:08.219118 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:08.219577 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:08.219578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:08.219579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:08.219649 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:08.219942 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:08.219943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:08.219944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:08.220105 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:08.220185 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:08.220357 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 16:49:08.220452 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 16:49:08.220588 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.212939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38001 (local address 127.0.98.3:47573)
0619 16:49:08.213095 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:08.213108 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:08.213175 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:08.213694 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:08.213695 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:08.213961 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:08.214049 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:08.214052 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.214958 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.214959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:08.215523 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:08.215525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.215933 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.215934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:08.215991 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:08.216613 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:08.216619 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:08.217138 (+ 519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:08.219217 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:08.219218 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:08.219219 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:08.219473 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:08.219732 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:08.219733 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:08.219733 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:08.219816 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:08.220176 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:08.220177 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:08.220237 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 16:49:08.220311 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 16:49:08.220464 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
Jun 19 16:49:08 dist-test-slave-jsw9 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887748, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 16:49:08 dist-test-slave-jsw9 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887748, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:08.262344 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.251477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50990 (local address 127.0.98.62:38001)
0619 16:49:08.251580 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:08.251583 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:08.251616 (+ 33us) server_negotiation.cc:407] Connection header received
0619 16:49:08.251667 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:08.251669 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:08.251687 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:08.251819 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:08.252684 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.253716 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.254877 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.255087 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.255254 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:08.258640 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:08.258643 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:08.258643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:08.258654 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:08.260326 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:08.261074 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:08.261080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:08.261081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:08.261169 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:08.261591 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:08.261592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:08.261592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:08.261746 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:08.261845 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:08.262081 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 16:49:08.262218 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:49:08.265249 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50990:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:49:08.267488 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:49:08.276055 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.270639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37749 (local address 127.0.0.1:59666)
0619 16:49:08.271024 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:08.271079 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:08.271175 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:08.271560 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:08.271565 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:08.271568 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:08.271705 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:08.271708 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.273341 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.273342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:08.274775 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:08.274779 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.275632 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.275634 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:08.275735 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:08.275783 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 16:49:08.275869 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 16:49:08.275942 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":36,"thread_start_us":71,"threads_started":1}
I20260619 16:49:08.276505 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.270794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59666 (local address 127.0.98.3:37749)
0619 16:49:08.271372 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:08.271377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:08.271388 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:49:08.271429 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:08.271431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:08.271458 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:08.271657 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:08.271788 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.273211 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.274915 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.275527 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.275621 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:08.275725 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:08.276334 (+ 609us) server_negotiation.cc:299] Negotiation successful
0619 16:49:08.276391 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":103,"threads_started":1}
I20260619 16:49:08.277015 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.270638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36867 (local address 127.0.0.1:50036)
0619 16:49:08.271021 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:08.271205 (+ 184us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:08.271276 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:08.271560 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:08.271565 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:08.271571 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:08.271685 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:08.271688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.274887 (+ 3199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.274888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:08.275622 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:08.275624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.276750 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.276751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:08.276811 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:08.276816 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:49:08.276874 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 16:49:08.276931 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":2,"spinlock_wait_cycles":768,"thread_start_us":125,"threads_started":1}
I20260619 16:49:08.277086 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.270805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50036 (local address 127.0.98.2:36867)
0619 16:49:08.271109 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:08.271115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:08.271329 (+ 214us) server_negotiation.cc:407] Connection header received
0619 16:49:08.271380 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:08.271383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:08.271402 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:08.271535 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:08.271772 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.273183 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.275721 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.276496 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.276569 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:08.276675 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:08.276915 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 16:49:08.276969 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":92,"threads_started":1}
I20260619 16:49:08.277232 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.271075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35848 (local address 127.0.98.1:34185)
0619 16:49:08.271801 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:08.271805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:08.271814 (+ 9us) server_negotiation.cc:407] Connection header received
0619 16:49:08.271851 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:08.271852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:08.271865 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:08.271935 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:08.272084 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.272869 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.275781 (+ 2912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.276595 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.276665 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:08.276746 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:08.277064 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 16:49:08.277127 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":114,"threads_started":1}
I20260619 16:49:08.277396 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.270877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34185 (local address 127.0.0.1:35848)
0619 16:49:08.271158 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:08.271167 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:08.271249 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:08.271944 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:08.271945 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:08.271946 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:08.272011 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:08.272013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.272990 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.272991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:08.274777 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:08.274779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.276782 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.276783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:08.276865 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:08.276871 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 16:49:08.276922 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 16:49:08.276985 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":140,"thread_start_us":69,"threads_started":1}
I20260619 16:49:08.277426 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 199f08ab92704ad8b22998ecb751ca00 (DEFAULT_TABLE table=test-table [id=77c53e9b2e0c49ca9200199c8210d9b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:08.277881 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 199f08ab92704ad8b22998ecb751ca00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:08.281801 2013 tablet_bootstrap.cc:492] T 199f08ab92704ad8b22998ecb751ca00 P 4ee5b1c2133f44d0b27c33bebfb6ad4b: Bootstrap starting.
I20260619 16:49:08.282829 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 199f08ab92704ad8b22998ecb751ca00 (DEFAULT_TABLE table=test-table [id=77c53e9b2e0c49ca9200199c8210d9b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:08.282727 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 199f08ab92704ad8b22998ecb751ca00 (DEFAULT_TABLE table=test-table [id=77c53e9b2e0c49ca9200199c8210d9b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:08.283151 2013 tablet_bootstrap.cc:654] T 199f08ab92704ad8b22998ecb751ca00 P 4ee5b1c2133f44d0b27c33bebfb6ad4b: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:08.283232 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 199f08ab92704ad8b22998ecb751ca00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:08.283231 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 199f08ab92704ad8b22998ecb751ca00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:08.283490 2013 log.cc:824] T 199f08ab92704ad8b22998ecb751ca00 P 4ee5b1c2133f44d0b27c33bebfb6ad4b: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:08.284363 2013 tablet_bootstrap.cc:492] T 199f08ab92704ad8b22998ecb751ca00 P 4ee5b1c2133f44d0b27c33bebfb6ad4b: No bootstrap required, opened a new log
I20260619 16:49:08.284453 2013 ts_tablet_manager.cc:1397] T 199f08ab92704ad8b22998ecb751ca00 P 4ee5b1c2133f44d0b27c33bebfb6ad4b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:49:08.286135 2013 raft_consensus.cc:348] T 199f08ab92704ad8b22998ecb751ca00 P 4ee5b1c2133f44d0b27c33bebfb6ad4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } } peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } }
I20260619 16:49:08.286378 2013 raft_consensus.cc:374] T 199f08ab92704ad8b22998ecb751ca00 P 4ee5b1c2133f44d0b27c33bebfb6ad4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:08.286437 2013 raft_consensus.cc:729] T 199f08ab92704ad8b22998ecb751ca00 P 4ee5b1c2133f44d0b27c33bebfb6ad4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ee5b1c2133f44d0b27c33bebfb6ad4b, State: Initialized, Role: FOLLOWER
I20260619 16:49:08.286619 2013 consensus_queue.cc:260] T 199f08ab92704ad8b22998ecb751ca00 P 4ee5b1c2133f44d0b27c33bebfb6ad4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } } peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } }
I20260619 16:49:08.286995 2013 ts_tablet_manager.cc:1428] T 199f08ab92704ad8b22998ecb751ca00 P 4ee5b1c2133f44d0b27c33bebfb6ad4b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 16:49:08.287173 1990 heartbeater.cc:503] Master 127.0.98.62:38001 was elected leader, sending a full tablet report...
I20260619 16:49:08.288478 2016 tablet_bootstrap.cc:492] T 199f08ab92704ad8b22998ecb751ca00 P 8f265f16f78f459fa0bf39bec82cee7d: Bootstrap starting.
I20260619 16:49:08.288882 2017 tablet_bootstrap.cc:492] T 199f08ab92704ad8b22998ecb751ca00 P 6371d7f13b834b088a3e199b2ad9c1da: Bootstrap starting.
I20260619 16:49:08.289321 2016 tablet_bootstrap.cc:654] T 199f08ab92704ad8b22998ecb751ca00 P 8f265f16f78f459fa0bf39bec82cee7d: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:08.289485 2017 tablet_bootstrap.cc:654] T 199f08ab92704ad8b22998ecb751ca00 P 6371d7f13b834b088a3e199b2ad9c1da: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:08.289683 2016 log.cc:824] T 199f08ab92704ad8b22998ecb751ca00 P 8f265f16f78f459fa0bf39bec82cee7d: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:08.289882 2017 log.cc:824] T 199f08ab92704ad8b22998ecb751ca00 P 6371d7f13b834b088a3e199b2ad9c1da: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:08.290670 2016 tablet_bootstrap.cc:492] T 199f08ab92704ad8b22998ecb751ca00 P 8f265f16f78f459fa0bf39bec82cee7d: No bootstrap required, opened a new log
I20260619 16:49:08.290746 2017 tablet_bootstrap.cc:492] T 199f08ab92704ad8b22998ecb751ca00 P 6371d7f13b834b088a3e199b2ad9c1da: No bootstrap required, opened a new log
I20260619 16:49:08.290746 2016 ts_tablet_manager.cc:1397] T 199f08ab92704ad8b22998ecb751ca00 P 8f265f16f78f459fa0bf39bec82cee7d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:49:08.290822 2017 ts_tablet_manager.cc:1397] T 199f08ab92704ad8b22998ecb751ca00 P 6371d7f13b834b088a3e199b2ad9c1da: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:49:08.293015 2017 raft_consensus.cc:348] T 199f08ab92704ad8b22998ecb751ca00 P 6371d7f13b834b088a3e199b2ad9c1da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } } peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } }
I20260619 16:49:08.293269 2016 raft_consensus.cc:348] T 199f08ab92704ad8b22998ecb751ca00 P 8f265f16f78f459fa0bf39bec82cee7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } } peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } }
I20260619 16:49:08.293354 2017 raft_consensus.cc:374] T 199f08ab92704ad8b22998ecb751ca00 P 6371d7f13b834b088a3e199b2ad9c1da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:08.293442 2016 raft_consensus.cc:374] T 199f08ab92704ad8b22998ecb751ca00 P 8f265f16f78f459fa0bf39bec82cee7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:08.293462 2017 raft_consensus.cc:729] T 199f08ab92704ad8b22998ecb751ca00 P 6371d7f13b834b088a3e199b2ad9c1da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6371d7f13b834b088a3e199b2ad9c1da, State: Initialized, Role: FOLLOWER
I20260619 16:49:08.293488 2016 raft_consensus.cc:729] T 199f08ab92704ad8b22998ecb751ca00 P 8f265f16f78f459fa0bf39bec82cee7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f265f16f78f459fa0bf39bec82cee7d, State: Initialized, Role: FOLLOWER
I20260619 16:49:08.293624 2016 consensus_queue.cc:260] T 199f08ab92704ad8b22998ecb751ca00 P 8f265f16f78f459fa0bf39bec82cee7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } } peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } }
I20260619 16:49:08.293625 2017 consensus_queue.cc:260] T 199f08ab92704ad8b22998ecb751ca00 P 6371d7f13b834b088a3e199b2ad9c1da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } } peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } }
I20260619 16:49:08.293993 2017 ts_tablet_manager.cc:1428] T 199f08ab92704ad8b22998ecb751ca00 P 6371d7f13b834b088a3e199b2ad9c1da: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 16:49:08.294045 1850 heartbeater.cc:503] Master 127.0.98.62:38001 was elected leader, sending a full tablet report...
I20260619 16:49:08.293995 2016 ts_tablet_manager.cc:1428] T 199f08ab92704ad8b22998ecb751ca00 P 8f265f16f78f459fa0bf39bec82cee7d: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260619 16:49:08.294111 1710 heartbeater.cc:503] Master 127.0.98.62:38001 was elected leader, sending a full tablet report...
I20260619 16:49:08.312294 2021 raft_consensus.cc:482] T 199f08ab92704ad8b22998ecb751ca00 P 8f265f16f78f459fa0bf39bec82cee7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:08.312479 2021 raft_consensus.cc:504] T 199f08ab92704ad8b22998ecb751ca00 P 8f265f16f78f459fa0bf39bec82cee7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } } peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } }
I20260619 16:49:08.313318 2021 leader_election.cc:288] T 199f08ab92704ad8b22998ecb751ca00 P 8f265f16f78f459fa0bf39bec82cee7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6371d7f13b834b088a3e199b2ad9c1da (127.0.98.2:36867), 4ee5b1c2133f44d0b27c33bebfb6ad4b (127.0.98.3:37749)
I20260619 16:49:08.318235 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.313751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36867 (local address 127.0.98.1:49939)
0619 16:49:08.313833 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:08.313847 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:08.313906 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:08.314165 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:08.314166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:08.314168 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:08.314295 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:08.314298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.315365 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.315366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:08.316902 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:08.316909 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.317855 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.317856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:08.317943 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:08.317947 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:49:08.318031 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 16:49:08.318106 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":103}
I20260619 16:49:08.318235 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.313867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49939 (local address 127.0.98.2:36867)
0619 16:49:08.313988 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:08.313990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:08.314000 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:49:08.314033 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:08.314034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:08.314048 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:08.314101 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:08.314376 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.315211 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.317073 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.317682 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.317768 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:08.317782 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:08.318050 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 16:49:08.318106 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260619 16:49:08.318677 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.313798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37749 (local address 127.0.98.1:56877)
0619 16:49:08.314035 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:08.314044 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:08.314099 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:08.314334 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:08.314336 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:08.314337 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:08.314403 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:08.314404 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.315365 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.315366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:08.316902 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:08.316909 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.318393 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.318394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:08.318472 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:08.318476 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:49:08.318528 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 16:49:08.318573 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":16,"thread_start_us":180,"threads_started":1}
I20260619 16:49:08.318689 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.314098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56877 (local address 127.0.98.3:37749)
0619 16:49:08.314175 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:08.314177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:08.314210 (+ 33us) server_negotiation.cc:407] Connection header received
0619 16:49:08.314247 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:08.314248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:08.314261 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:08.314322 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:08.314465 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.315214 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.317074 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.317682 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.317768 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:08.317782 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:08.318558 (+ 776us) server_negotiation.cc:299] Negotiation successful
0619 16:49:08.318620 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 16:49:08.319025 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "199f08ab92704ad8b22998ecb751ca00" candidate_uuid: "8f265f16f78f459fa0bf39bec82cee7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" is_pre_election: true
I20260619 16:49:08.318836 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "199f08ab92704ad8b22998ecb751ca00" candidate_uuid: "8f265f16f78f459fa0bf39bec82cee7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6371d7f13b834b088a3e199b2ad9c1da" is_pre_election: true
I20260619 16:49:08.319191 1944 raft_consensus.cc:2454] T 199f08ab92704ad8b22998ecb751ca00 P 4ee5b1c2133f44d0b27c33bebfb6ad4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f265f16f78f459fa0bf39bec82cee7d in term 0.
I20260619 16:49:08.319280 1804 raft_consensus.cc:2454] T 199f08ab92704ad8b22998ecb751ca00 P 6371d7f13b834b088a3e199b2ad9c1da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f265f16f78f459fa0bf39bec82cee7d in term 0.
I20260619 16:49:08.319483 1596 leader_election.cc:302] T 199f08ab92704ad8b22998ecb751ca00 P 8f265f16f78f459fa0bf39bec82cee7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ee5b1c2133f44d0b27c33bebfb6ad4b, 8f265f16f78f459fa0bf39bec82cee7d; no voters:
I20260619 16:49:08.319648 2021 raft_consensus.cc:2781] T 199f08ab92704ad8b22998ecb751ca00 P 8f265f16f78f459fa0bf39bec82cee7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:49:08.319713 2021 raft_consensus.cc:482] T 199f08ab92704ad8b22998ecb751ca00 P 8f265f16f78f459fa0bf39bec82cee7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:49:08.319737 2021 raft_consensus.cc:3037] T 199f08ab92704ad8b22998ecb751ca00 P 8f265f16f78f459fa0bf39bec82cee7d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:08.320608 2021 raft_consensus.cc:504] T 199f08ab92704ad8b22998ecb751ca00 P 8f265f16f78f459fa0bf39bec82cee7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } } peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } }
I20260619 16:49:08.320778 2021 leader_election.cc:288] T 199f08ab92704ad8b22998ecb751ca00 P 8f265f16f78f459fa0bf39bec82cee7d [CANDIDATE]: Term 1 election: Requested vote from peers 6371d7f13b834b088a3e199b2ad9c1da (127.0.98.2:36867), 4ee5b1c2133f44d0b27c33bebfb6ad4b (127.0.98.3:37749)
I20260619 16:49:08.321007 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "199f08ab92704ad8b22998ecb751ca00" candidate_uuid: "8f265f16f78f459fa0bf39bec82cee7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b"
I20260619 16:49:08.321002 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "199f08ab92704ad8b22998ecb751ca00" candidate_uuid: "8f265f16f78f459fa0bf39bec82cee7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6371d7f13b834b088a3e199b2ad9c1da"
I20260619 16:49:08.321101 1804 raft_consensus.cc:3037] T 199f08ab92704ad8b22998ecb751ca00 P 6371d7f13b834b088a3e199b2ad9c1da [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:08.321099 1944 raft_consensus.cc:3037] T 199f08ab92704ad8b22998ecb751ca00 P 4ee5b1c2133f44d0b27c33bebfb6ad4b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:08.322006 1804 raft_consensus.cc:2454] T 199f08ab92704ad8b22998ecb751ca00 P 6371d7f13b834b088a3e199b2ad9c1da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f265f16f78f459fa0bf39bec82cee7d in term 1.
I20260619 16:49:08.322562 1595 leader_election.cc:302] T 199f08ab92704ad8b22998ecb751ca00 P 8f265f16f78f459fa0bf39bec82cee7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6371d7f13b834b088a3e199b2ad9c1da, 8f265f16f78f459fa0bf39bec82cee7d; no voters:
I20260619 16:49:08.322670 2021 raft_consensus.cc:2781] T 199f08ab92704ad8b22998ecb751ca00 P 8f265f16f78f459fa0bf39bec82cee7d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:08.322840 1944 raft_consensus.cc:2454] T 199f08ab92704ad8b22998ecb751ca00 P 4ee5b1c2133f44d0b27c33bebfb6ad4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f265f16f78f459fa0bf39bec82cee7d in term 1.
I20260619 16:49:08.322881 2021 raft_consensus.cc:686] T 199f08ab92704ad8b22998ecb751ca00 P 8f265f16f78f459fa0bf39bec82cee7d [term 1 LEADER]: Becoming Leader. State: Replica: 8f265f16f78f459fa0bf39bec82cee7d, State: Running, Role: LEADER
I20260619 16:49:08.323006 2021 consensus_queue.cc:237] T 199f08ab92704ad8b22998ecb751ca00 P 8f265f16f78f459fa0bf39bec82cee7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } } peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } }
I20260619 16:49:08.325114 1507 catalog_manager.cc:5495] T 199f08ab92704ad8b22998ecb751ca00 P 8f265f16f78f459fa0bf39bec82cee7d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f265f16f78f459fa0bf39bec82cee7d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8f265f16f78f459fa0bf39bec82cee7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } health_report { overall_health: HEALTHY } } }
W20260619 16:49:08.333181 1711 tablet.cc:2369] T 199f08ab92704ad8b22998ecb751ca00 P 8f265f16f78f459fa0bf39bec82cee7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:49:08.337841 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.334252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35862 (local address 127.0.98.1:34185)
0619 16:49:08.334310 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:08.334312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:08.334321 (+ 9us) server_negotiation.cc:407] Connection header received
0619 16:49:08.334421 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:08.334422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:08.334443 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:08.334497 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:08.334779 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.335487 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.336619 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.336827 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.337004 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:08.337357 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:08.337483 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:08.337562 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:08.337698 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 16:49:08.337729 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 16:49:08.340282 1944 raft_consensus.cc:1261] T 199f08ab92704ad8b22998ecb751ca00 P 4ee5b1c2133f44d0b27c33bebfb6ad4b [term 1 FOLLOWER]: Refusing update from remote peer 8f265f16f78f459fa0bf39bec82cee7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:08.340282 1804 raft_consensus.cc:1261] T 199f08ab92704ad8b22998ecb751ca00 P 6371d7f13b834b088a3e199b2ad9c1da [term 1 FOLLOWER]: Refusing update from remote peer 8f265f16f78f459fa0bf39bec82cee7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:08.340701 2023 consensus_queue.cc:1035] T 199f08ab92704ad8b22998ecb751ca00 P 8f265f16f78f459fa0bf39bec82cee7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:08.340888 2021 consensus_queue.cc:1035] T 199f08ab92704ad8b22998ecb751ca00 P 8f265f16f78f459fa0bf39bec82cee7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:08.345230 2027 mvcc.cc:205] Tried to move back new op lower bound from 7298612217198702592 to 7298612217132687360. Current Snapshot: MvccSnapshot[applied={T|T < 7298612217198702592}]
I20260619 16:49:08.344790 2028 mvcc.cc:205] Tried to move back new op lower bound from 7298612217198702592 to 7298612217132687360. Current Snapshot: MvccSnapshot[applied={T|T < 7298612217198702592 or (T in {7298612217198702592})}]
I20260619 16:49:08.348731 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:08.348871 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:08.349709 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7ff670f5ffd941f086bf2f9c2d9955d1: Sending DeleteTablet for 3 replicas of tablet 199f08ab92704ad8b22998ecb751ca00
I20260619 16:49:08.350131 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 199f08ab92704ad8b22998ecb751ca00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59666
I20260619 16:49:08.350159 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 199f08ab92704ad8b22998ecb751ca00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35848
I20260619 16:49:08.350167 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 199f08ab92704ad8b22998ecb751ca00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50036
I20260619 16:49:08.350445 2037 tablet_replica.cc:330] T 199f08ab92704ad8b22998ecb751ca00 P 8f265f16f78f459fa0bf39bec82cee7d: stopping tablet replica
I20260619 16:49:08.350557 2037 raft_consensus.cc:2229] T 199f08ab92704ad8b22998ecb751ca00 P 8f265f16f78f459fa0bf39bec82cee7d [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:49:08.350683 2037 raft_consensus.cc:2258] T 199f08ab92704ad8b22998ecb751ca00 P 8f265f16f78f459fa0bf39bec82cee7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:08.350879 2038 tablet_replica.cc:330] T 199f08ab92704ad8b22998ecb751ca00 P 4ee5b1c2133f44d0b27c33bebfb6ad4b: stopping tablet replica
I20260619 16:49:08.351004 2038 raft_consensus.cc:2229] T 199f08ab92704ad8b22998ecb751ca00 P 4ee5b1c2133f44d0b27c33bebfb6ad4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:08.350934 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50990:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:49:08.351110 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:49:08.351408 2038 raft_consensus.cc:2258] T 199f08ab92704ad8b22998ecb751ca00 P 4ee5b1c2133f44d0b27c33bebfb6ad4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:08.351568 2037 ts_tablet_manager.cc:1905] T 199f08ab92704ad8b22998ecb751ca00 P 8f265f16f78f459fa0bf39bec82cee7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:08.352576 2038 ts_tablet_manager.cc:1905] T 199f08ab92704ad8b22998ecb751ca00 P 4ee5b1c2133f44d0b27c33bebfb6ad4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:08.352702 1924 tablet_service.cc:1467] Processing CreateTablet for tablet f9a5c735de6f4b54b1fe33c939fb6413 (DEFAULT_TABLE table=test-table [id=9e30ecd4082e46918aa61191d7fe0d8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:08.352800 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9a5c735de6f4b54b1fe33c939fb6413. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:08.353590 2037 ts_tablet_manager.cc:1918] T 199f08ab92704ad8b22998ecb751ca00 P 8f265f16f78f459fa0bf39bec82cee7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:08.353667 2037 log.cc:1196] T 199f08ab92704ad8b22998ecb751ca00 P 8f265f16f78f459fa0bf39bec82cee7d: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-0/wal/wals/199f08ab92704ad8b22998ecb751ca00
I20260619 16:49:08.353955 2037 ts_tablet_manager.cc:1939] T 199f08ab92704ad8b22998ecb751ca00 P 8f265f16f78f459fa0bf39bec82cee7d: Deleting consensus metadata
I20260619 16:49:08.354598 2013 tablet_bootstrap.cc:492] T f9a5c735de6f4b54b1fe33c939fb6413 P 4ee5b1c2133f44d0b27c33bebfb6ad4b: Bootstrap starting.
I20260619 16:49:08.354657 1492 catalog_manager.cc:4841] TS 8f265f16f78f459fa0bf39bec82cee7d (127.0.98.1:34185): tablet 199f08ab92704ad8b22998ecb751ca00 (table test-table [id=77c53e9b2e0c49ca9200199c8210d9b9]) successfully deleted
I20260619 16:49:08.354882 2038 ts_tablet_manager.cc:1918] T 199f08ab92704ad8b22998ecb751ca00 P 4ee5b1c2133f44d0b27c33bebfb6ad4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:08.354955 2038 log.cc:1196] T 199f08ab92704ad8b22998ecb751ca00 P 4ee5b1c2133f44d0b27c33bebfb6ad4b: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-2/wal/wals/199f08ab92704ad8b22998ecb751ca00
I20260619 16:49:08.355079 2013 tablet_bootstrap.cc:654] T f9a5c735de6f4b54b1fe33c939fb6413 P 4ee5b1c2133f44d0b27c33bebfb6ad4b: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:08.355207 2038 ts_tablet_manager.cc:1939] T 199f08ab92704ad8b22998ecb751ca00 P 4ee5b1c2133f44d0b27c33bebfb6ad4b: Deleting consensus metadata
I20260619 16:49:08.355432 2039 tablet_replica.cc:330] T 199f08ab92704ad8b22998ecb751ca00 P 6371d7f13b834b088a3e199b2ad9c1da: stopping tablet replica
I20260619 16:49:08.355792 2013 tablet_bootstrap.cc:492] T f9a5c735de6f4b54b1fe33c939fb6413 P 4ee5b1c2133f44d0b27c33bebfb6ad4b: No bootstrap required, opened a new log
I20260619 16:49:08.355840 2013 ts_tablet_manager.cc:1397] T f9a5c735de6f4b54b1fe33c939fb6413 P 4ee5b1c2133f44d0b27c33bebfb6ad4b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:08.355914 2039 raft_consensus.cc:2229] T 199f08ab92704ad8b22998ecb751ca00 P 6371d7f13b834b088a3e199b2ad9c1da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:08.356017 1784 tablet_service.cc:1467] Processing CreateTablet for tablet f9a5c735de6f4b54b1fe33c939fb6413 (DEFAULT_TABLE table=test-table [id=9e30ecd4082e46918aa61191d7fe0d8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:08.356061 2039 raft_consensus.cc:2258] T 199f08ab92704ad8b22998ecb751ca00 P 6371d7f13b834b088a3e199b2ad9c1da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:08.356019 2013 raft_consensus.cc:348] T f9a5c735de6f4b54b1fe33c939fb6413 P 4ee5b1c2133f44d0b27c33bebfb6ad4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } } peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } }
I20260619 16:49:08.356103 2013 raft_consensus.cc:374] T f9a5c735de6f4b54b1fe33c939fb6413 P 4ee5b1c2133f44d0b27c33bebfb6ad4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:08.356125 2013 raft_consensus.cc:729] T f9a5c735de6f4b54b1fe33c939fb6413 P 4ee5b1c2133f44d0b27c33bebfb6ad4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ee5b1c2133f44d0b27c33bebfb6ad4b, State: Initialized, Role: FOLLOWER
I20260619 16:49:08.356137 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9a5c735de6f4b54b1fe33c939fb6413. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:08.356181 2013 consensus_queue.cc:260] T f9a5c735de6f4b54b1fe33c939fb6413 P 4ee5b1c2133f44d0b27c33bebfb6ad4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } } peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } }
I20260619 16:49:08.356276 2013 ts_tablet_manager.cc:1428] T f9a5c735de6f4b54b1fe33c939fb6413 P 4ee5b1c2133f44d0b27c33bebfb6ad4b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 16:49:08.357606 2017 tablet_bootstrap.cc:492] T f9a5c735de6f4b54b1fe33c939fb6413 P 6371d7f13b834b088a3e199b2ad9c1da: Bootstrap starting.
I20260619 16:49:08.357695 1644 tablet_service.cc:1467] Processing CreateTablet for tablet f9a5c735de6f4b54b1fe33c939fb6413 (DEFAULT_TABLE table=test-table [id=9e30ecd4082e46918aa61191d7fe0d8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:08.357816 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9a5c735de6f4b54b1fe33c939fb6413. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:08.358139 2017 tablet_bootstrap.cc:654] T f9a5c735de6f4b54b1fe33c939fb6413 P 6371d7f13b834b088a3e199b2ad9c1da: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:08.358901 2017 tablet_bootstrap.cc:492] T f9a5c735de6f4b54b1fe33c939fb6413 P 6371d7f13b834b088a3e199b2ad9c1da: No bootstrap required, opened a new log
I20260619 16:49:08.358942 2017 ts_tablet_manager.cc:1397] T f9a5c735de6f4b54b1fe33c939fb6413 P 6371d7f13b834b088a3e199b2ad9c1da: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:08.359107 2017 raft_consensus.cc:348] T f9a5c735de6f4b54b1fe33c939fb6413 P 6371d7f13b834b088a3e199b2ad9c1da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } } peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } }
I20260619 16:49:08.359179 2017 raft_consensus.cc:374] T f9a5c735de6f4b54b1fe33c939fb6413 P 6371d7f13b834b088a3e199b2ad9c1da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:08.359200 2017 raft_consensus.cc:729] T f9a5c735de6f4b54b1fe33c939fb6413 P 6371d7f13b834b088a3e199b2ad9c1da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6371d7f13b834b088a3e199b2ad9c1da, State: Initialized, Role: FOLLOWER
I20260619 16:49:08.359259 2017 consensus_queue.cc:260] T f9a5c735de6f4b54b1fe33c939fb6413 P 6371d7f13b834b088a3e199b2ad9c1da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } } peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } }
I20260619 16:49:08.359354 2017 ts_tablet_manager.cc:1428] T f9a5c735de6f4b54b1fe33c939fb6413 P 6371d7f13b834b088a3e199b2ad9c1da: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 16:49:08.359436 2016 tablet_bootstrap.cc:492] T f9a5c735de6f4b54b1fe33c939fb6413 P 8f265f16f78f459fa0bf39bec82cee7d: Bootstrap starting.
I20260619 16:49:08.359925 2016 tablet_bootstrap.cc:654] T f9a5c735de6f4b54b1fe33c939fb6413 P 8f265f16f78f459fa0bf39bec82cee7d: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:08.361416 1492 catalog_manager.cc:4841] TS 4ee5b1c2133f44d0b27c33bebfb6ad4b (127.0.98.3:37749): tablet 199f08ab92704ad8b22998ecb751ca00 (table test-table [id=77c53e9b2e0c49ca9200199c8210d9b9]) successfully deleted
I20260619 16:49:08.361502 2016 tablet_bootstrap.cc:492] T f9a5c735de6f4b54b1fe33c939fb6413 P 8f265f16f78f459fa0bf39bec82cee7d: No bootstrap required, opened a new log
I20260619 16:49:08.361555 2016 ts_tablet_manager.cc:1397] T f9a5c735de6f4b54b1fe33c939fb6413 P 8f265f16f78f459fa0bf39bec82cee7d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:49:08.361714 2016 raft_consensus.cc:348] T f9a5c735de6f4b54b1fe33c939fb6413 P 8f265f16f78f459fa0bf39bec82cee7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } } peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } }
I20260619 16:49:08.361791 2016 raft_consensus.cc:374] T f9a5c735de6f4b54b1fe33c939fb6413 P 8f265f16f78f459fa0bf39bec82cee7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:08.361809 2016 raft_consensus.cc:729] T f9a5c735de6f4b54b1fe33c939fb6413 P 8f265f16f78f459fa0bf39bec82cee7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f265f16f78f459fa0bf39bec82cee7d, State: Initialized, Role: FOLLOWER
I20260619 16:49:08.361855 2016 consensus_queue.cc:260] T f9a5c735de6f4b54b1fe33c939fb6413 P 8f265f16f78f459fa0bf39bec82cee7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } } peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } }
I20260619 16:49:08.361953 2016 ts_tablet_manager.cc:1428] T f9a5c735de6f4b54b1fe33c939fb6413 P 8f265f16f78f459fa0bf39bec82cee7d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 16:49:08.362107 2039 ts_tablet_manager.cc:1905] T 199f08ab92704ad8b22998ecb751ca00 P 6371d7f13b834b088a3e199b2ad9c1da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:08.364212 2039 ts_tablet_manager.cc:1918] T 199f08ab92704ad8b22998ecb751ca00 P 6371d7f13b834b088a3e199b2ad9c1da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:08.364320 2039 log.cc:1196] T 199f08ab92704ad8b22998ecb751ca00 P 6371d7f13b834b088a3e199b2ad9c1da: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-1/wal/wals/199f08ab92704ad8b22998ecb751ca00
I20260619 16:49:08.364600 2039 ts_tablet_manager.cc:1939] T 199f08ab92704ad8b22998ecb751ca00 P 6371d7f13b834b088a3e199b2ad9c1da: Deleting consensus metadata
I20260619 16:49:08.365105 1491 catalog_manager.cc:4841] TS 6371d7f13b834b088a3e199b2ad9c1da (127.0.98.2:36867): tablet 199f08ab92704ad8b22998ecb751ca00 (table test-table [id=77c53e9b2e0c49ca9200199c8210d9b9]) successfully deleted
W20260619 16:49:08.457459 1991 tablet.cc:2369] T f9a5c735de6f4b54b1fe33c939fb6413 P 4ee5b1c2133f44d0b27c33bebfb6ad4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:49:08.518435 2015 raft_consensus.cc:482] T f9a5c735de6f4b54b1fe33c939fb6413 P 4ee5b1c2133f44d0b27c33bebfb6ad4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:08.518580 2015 raft_consensus.cc:504] T f9a5c735de6f4b54b1fe33c939fb6413 P 4ee5b1c2133f44d0b27c33bebfb6ad4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } } peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } }
I20260619 16:49:08.519080 2015 leader_election.cc:288] T f9a5c735de6f4b54b1fe33c939fb6413 P 4ee5b1c2133f44d0b27c33bebfb6ad4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f265f16f78f459fa0bf39bec82cee7d (127.0.98.1:34185), 6371d7f13b834b088a3e199b2ad9c1da (127.0.98.2:36867)
W20260619 16:49:08.521463 1851 tablet.cc:2369] T f9a5c735de6f4b54b1fe33c939fb6413 P 6371d7f13b834b088a3e199b2ad9c1da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:49:08.522714 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.519562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45637 (local address 127.0.98.2:36867)
0619 16:49:08.519615 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:08.519617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:08.519627 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:49:08.519658 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:08.519659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:08.519670 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:08.519720 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:08.519909 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.520457 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.521889 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.522358 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.522427 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:08.522440 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:08.522589 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 16:49:08.522641 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 16:49:08.522728 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.519361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36867 (local address 127.0.98.3:45637)
0619 16:49:08.519479 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:08.519501 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:08.519578 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:08.519735 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:08.519736 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:08.519738 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:08.519829 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:08.519831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.520552 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.520553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:08.521758 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:08.521762 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.522440 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.522441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:08.522516 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:08.522519 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:49:08.522565 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 16:49:08.522619 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":95}
I20260619 16:49:08.522713 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.519510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60931 (local address 127.0.98.1:34185)
0619 16:49:08.519621 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:08.519623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:08.519832 (+ 209us) server_negotiation.cc:407] Connection header received
0619 16:49:08.519892 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:08.519893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:08.519904 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:08.519955 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:08.520118 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.520640 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.521834 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.522315 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.522384 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:08.522409 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:08.522573 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 16:49:08.522625 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 16:49:08.522745 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.519407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34185 (local address 127.0.98.3:60931)
0619 16:49:08.519749 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:08.519759 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:08.519833 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:08.519958 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:08.519959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:08.519961 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:08.520053 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:08.520055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.520742 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.520743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:08.521731 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:08.521737 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.522429 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.522431 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:08.522502 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:08.522507 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:49:08.522555 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 16:49:08.522612 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":191,"thread_start_us":80,"threads_started":1}
I20260619 16:49:08.523033 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9a5c735de6f4b54b1fe33c939fb6413" candidate_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6371d7f13b834b088a3e199b2ad9c1da" is_pre_election: true
I20260619 16:49:08.523072 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9a5c735de6f4b54b1fe33c939fb6413" candidate_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f265f16f78f459fa0bf39bec82cee7d" is_pre_election: true
I20260619 16:49:08.523223 1804 raft_consensus.cc:2454] T f9a5c735de6f4b54b1fe33c939fb6413 P 6371d7f13b834b088a3e199b2ad9c1da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ee5b1c2133f44d0b27c33bebfb6ad4b in term 0.
I20260619 16:49:08.523241 1664 raft_consensus.cc:2454] T f9a5c735de6f4b54b1fe33c939fb6413 P 8f265f16f78f459fa0bf39bec82cee7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ee5b1c2133f44d0b27c33bebfb6ad4b in term 0.
I20260619 16:49:08.523459 1876 leader_election.cc:302] T f9a5c735de6f4b54b1fe33c939fb6413 P 4ee5b1c2133f44d0b27c33bebfb6ad4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ee5b1c2133f44d0b27c33bebfb6ad4b, 8f265f16f78f459fa0bf39bec82cee7d; no voters:
I20260619 16:49:08.523648 2015 raft_consensus.cc:2781] T f9a5c735de6f4b54b1fe33c939fb6413 P 4ee5b1c2133f44d0b27c33bebfb6ad4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:49:08.523701 2015 raft_consensus.cc:482] T f9a5c735de6f4b54b1fe33c939fb6413 P 4ee5b1c2133f44d0b27c33bebfb6ad4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:49:08.523717 2015 raft_consensus.cc:3037] T f9a5c735de6f4b54b1fe33c939fb6413 P 4ee5b1c2133f44d0b27c33bebfb6ad4b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:08.524741 2015 raft_consensus.cc:504] T f9a5c735de6f4b54b1fe33c939fb6413 P 4ee5b1c2133f44d0b27c33bebfb6ad4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } } peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } }
I20260619 16:49:08.524938 2015 leader_election.cc:288] T f9a5c735de6f4b54b1fe33c939fb6413 P 4ee5b1c2133f44d0b27c33bebfb6ad4b [CANDIDATE]: Term 1 election: Requested vote from peers 8f265f16f78f459fa0bf39bec82cee7d (127.0.98.1:34185), 6371d7f13b834b088a3e199b2ad9c1da (127.0.98.2:36867)
I20260619 16:49:08.525168 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9a5c735de6f4b54b1fe33c939fb6413" candidate_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6371d7f13b834b088a3e199b2ad9c1da"
I20260619 16:49:08.525285 1804 raft_consensus.cc:3037] T f9a5c735de6f4b54b1fe33c939fb6413 P 6371d7f13b834b088a3e199b2ad9c1da [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:08.525292 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9a5c735de6f4b54b1fe33c939fb6413" candidate_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f265f16f78f459fa0bf39bec82cee7d"
I20260619 16:49:08.525411 1664 raft_consensus.cc:3037] T f9a5c735de6f4b54b1fe33c939fb6413 P 8f265f16f78f459fa0bf39bec82cee7d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:08.526125 1804 raft_consensus.cc:2454] T f9a5c735de6f4b54b1fe33c939fb6413 P 6371d7f13b834b088a3e199b2ad9c1da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ee5b1c2133f44d0b27c33bebfb6ad4b in term 1.
I20260619 16:49:08.526419 1875 leader_election.cc:302] T f9a5c735de6f4b54b1fe33c939fb6413 P 4ee5b1c2133f44d0b27c33bebfb6ad4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ee5b1c2133f44d0b27c33bebfb6ad4b, 6371d7f13b834b088a3e199b2ad9c1da; no voters:
I20260619 16:49:08.526433 1664 raft_consensus.cc:2454] T f9a5c735de6f4b54b1fe33c939fb6413 P 8f265f16f78f459fa0bf39bec82cee7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ee5b1c2133f44d0b27c33bebfb6ad4b in term 1.
I20260619 16:49:08.526552 2015 raft_consensus.cc:2781] T f9a5c735de6f4b54b1fe33c939fb6413 P 4ee5b1c2133f44d0b27c33bebfb6ad4b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:08.526751 2015 raft_consensus.cc:686] T f9a5c735de6f4b54b1fe33c939fb6413 P 4ee5b1c2133f44d0b27c33bebfb6ad4b [term 1 LEADER]: Becoming Leader. State: Replica: 4ee5b1c2133f44d0b27c33bebfb6ad4b, State: Running, Role: LEADER
I20260619 16:49:08.526839 2015 consensus_queue.cc:237] T f9a5c735de6f4b54b1fe33c939fb6413 P 4ee5b1c2133f44d0b27c33bebfb6ad4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } } peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } }
I20260619 16:49:08.527931 1506 catalog_manager.cc:5495] T f9a5c735de6f4b54b1fe33c939fb6413 P 4ee5b1c2133f44d0b27c33bebfb6ad4b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ee5b1c2133f44d0b27c33bebfb6ad4b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } health_report { overall_health: HEALTHY } } }
Jun 19 16:49:08 dist-test-slave-jsw9 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:08.572158 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.563494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50992 (local address 127.0.98.62:38001)
0619 16:49:08.563633 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:08.563635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:08.563801 (+ 166us) server_negotiation.cc:407] Connection header received
0619 16:49:08.563889 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:08.563889 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:08.563901 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:08.563956 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:08.564605 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.565137 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.565882 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.566084 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.566148 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:08.569039 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:08.569040 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:08.569040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:08.569049 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:08.570833 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:08.571302 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:08.571304 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:08.571304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:08.571359 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:08.571542 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:08.571543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:08.571543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:08.571677 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:08.571748 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:08.571913 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 16:49:08.572018 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260619 16:49:08.572427 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.563369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38001 (local address 127.0.0.1:50992)
0619 16:49:08.563613 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:08.563631 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:08.563802 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:08.563965 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:08.563966 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:08.564399 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:08.564505 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:08.564509 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.565220 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.565221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:08.565787 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:08.565789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.566161 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.566162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:08.566273 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:08.566966 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:08.566971 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:08.568874 (+ 1903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:08.570945 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:08.570960 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:08.570964 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:08.571203 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:08.571423 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:08.571423 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:08.571424 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:08.571484 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:08.571744 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:08.571757 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:08.571808 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 16:49:08.571873 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 16:49:08.572130 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":106,"thread_start_us":115,"threads_started":1}
I20260619 16:49:08.576035 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50992:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 16:49:08.576364 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:49:08.578680 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 0495ad8bb97048ca893c91004aa30bae (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1d3fe25185314710a35498cdcd1f456b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:49:08.578866 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 0495ad8bb97048ca893c91004aa30bae (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1d3fe25185314710a35498cdcd1f456b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:49:08.578958 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0495ad8bb97048ca893c91004aa30bae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:08.579020 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0495ad8bb97048ca893c91004aa30bae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:08.580927 2013 tablet_bootstrap.cc:492] T 0495ad8bb97048ca893c91004aa30bae P 4ee5b1c2133f44d0b27c33bebfb6ad4b: Bootstrap starting.
I20260619 16:49:08.581089 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 0495ad8bb97048ca893c91004aa30bae (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1d3fe25185314710a35498cdcd1f456b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:49:08.581291 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0495ad8bb97048ca893c91004aa30bae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:08.581542 2013 tablet_bootstrap.cc:654] T 0495ad8bb97048ca893c91004aa30bae P 4ee5b1c2133f44d0b27c33bebfb6ad4b: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:08.582172 2017 tablet_bootstrap.cc:492] T 0495ad8bb97048ca893c91004aa30bae P 6371d7f13b834b088a3e199b2ad9c1da: Bootstrap starting.
I20260619 16:49:08.582748 2017 tablet_bootstrap.cc:654] T 0495ad8bb97048ca893c91004aa30bae P 6371d7f13b834b088a3e199b2ad9c1da: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:08.583214 2016 tablet_bootstrap.cc:492] T 0495ad8bb97048ca893c91004aa30bae P 8f265f16f78f459fa0bf39bec82cee7d: Bootstrap starting.
I20260619 16:49:08.583608 2016 tablet_bootstrap.cc:654] T 0495ad8bb97048ca893c91004aa30bae P 8f265f16f78f459fa0bf39bec82cee7d: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:08.583794 2017 tablet_bootstrap.cc:492] T 0495ad8bb97048ca893c91004aa30bae P 6371d7f13b834b088a3e199b2ad9c1da: No bootstrap required, opened a new log
I20260619 16:49:08.583849 2017 ts_tablet_manager.cc:1397] T 0495ad8bb97048ca893c91004aa30bae P 6371d7f13b834b088a3e199b2ad9c1da: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:49:08.583957 2013 tablet_bootstrap.cc:492] T 0495ad8bb97048ca893c91004aa30bae P 4ee5b1c2133f44d0b27c33bebfb6ad4b: No bootstrap required, opened a new log
I20260619 16:49:08.584048 2013 ts_tablet_manager.cc:1397] T 0495ad8bb97048ca893c91004aa30bae P 4ee5b1c2133f44d0b27c33bebfb6ad4b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 16:49:08.584043 2017 raft_consensus.cc:348] T 0495ad8bb97048ca893c91004aa30bae P 6371d7f13b834b088a3e199b2ad9c1da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } } peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } }
I20260619 16:49:08.584260 2017 raft_consensus.cc:374] T 0495ad8bb97048ca893c91004aa30bae P 6371d7f13b834b088a3e199b2ad9c1da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:08.584460 2017 raft_consensus.cc:729] T 0495ad8bb97048ca893c91004aa30bae P 6371d7f13b834b088a3e199b2ad9c1da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6371d7f13b834b088a3e199b2ad9c1da, State: Initialized, Role: FOLLOWER
I20260619 16:49:08.584523 2017 consensus_queue.cc:260] T 0495ad8bb97048ca893c91004aa30bae P 6371d7f13b834b088a3e199b2ad9c1da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } } peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } }
I20260619 16:49:08.584676 2017 ts_tablet_manager.cc:1428] T 0495ad8bb97048ca893c91004aa30bae P 6371d7f13b834b088a3e199b2ad9c1da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:08.584982 2013 raft_consensus.cc:348] T 0495ad8bb97048ca893c91004aa30bae P 4ee5b1c2133f44d0b27c33bebfb6ad4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } } peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } }
I20260619 16:49:08.585088 2013 raft_consensus.cc:374] T 0495ad8bb97048ca893c91004aa30bae P 4ee5b1c2133f44d0b27c33bebfb6ad4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:08.585114 2013 raft_consensus.cc:729] T 0495ad8bb97048ca893c91004aa30bae P 4ee5b1c2133f44d0b27c33bebfb6ad4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ee5b1c2133f44d0b27c33bebfb6ad4b, State: Initialized, Role: FOLLOWER
I20260619 16:49:08.585055 2020 tablet_replica.cc:439] T 0495ad8bb97048ca893c91004aa30bae P 6371d7f13b834b088a3e199b2ad9c1da: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } } peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } } }
I20260619 16:49:08.585150 2020 tablet_replica.cc:442] T 0495ad8bb97048ca893c91004aa30bae P 6371d7f13b834b088a3e199b2ad9c1da: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:08.585156 2013 consensus_queue.cc:260] T 0495ad8bb97048ca893c91004aa30bae P 4ee5b1c2133f44d0b27c33bebfb6ad4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } } peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } }
I20260619 16:49:08.585256 2013 ts_tablet_manager.cc:1428] T 0495ad8bb97048ca893c91004aa30bae P 4ee5b1c2133f44d0b27c33bebfb6ad4b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:08.585433 2015 tablet_replica.cc:439] T 0495ad8bb97048ca893c91004aa30bae P 4ee5b1c2133f44d0b27c33bebfb6ad4b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } } peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } } }
I20260619 16:49:08.585511 2015 tablet_replica.cc:442] T 0495ad8bb97048ca893c91004aa30bae P 4ee5b1c2133f44d0b27c33bebfb6ad4b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:08.585624 2016 tablet_bootstrap.cc:492] T 0495ad8bb97048ca893c91004aa30bae P 8f265f16f78f459fa0bf39bec82cee7d: No bootstrap required, opened a new log
I20260619 16:49:08.585683 2016 ts_tablet_manager.cc:1397] T 0495ad8bb97048ca893c91004aa30bae P 8f265f16f78f459fa0bf39bec82cee7d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:49:08.585865 2016 raft_consensus.cc:348] T 0495ad8bb97048ca893c91004aa30bae P 8f265f16f78f459fa0bf39bec82cee7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } } peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } }
I20260619 16:49:08.585952 2016 raft_consensus.cc:374] T 0495ad8bb97048ca893c91004aa30bae P 8f265f16f78f459fa0bf39bec82cee7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:08.585974 2016 raft_consensus.cc:729] T 0495ad8bb97048ca893c91004aa30bae P 8f265f16f78f459fa0bf39bec82cee7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f265f16f78f459fa0bf39bec82cee7d, State: Initialized, Role: FOLLOWER
I20260619 16:49:08.586027 2016 consensus_queue.cc:260] T 0495ad8bb97048ca893c91004aa30bae P 8f265f16f78f459fa0bf39bec82cee7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } } peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } }
I20260619 16:49:08.586129 2016 ts_tablet_manager.cc:1428] T 0495ad8bb97048ca893c91004aa30bae P 8f265f16f78f459fa0bf39bec82cee7d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:08.586226 2034 tablet_replica.cc:439] T 0495ad8bb97048ca893c91004aa30bae P 8f265f16f78f459fa0bf39bec82cee7d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } } peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } } }
I20260619 16:49:08.586338 2034 tablet_replica.cc:442] T 0495ad8bb97048ca893c91004aa30bae P 8f265f16f78f459fa0bf39bec82cee7d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:08.721050 2020 raft_consensus.cc:482] T 0495ad8bb97048ca893c91004aa30bae P 6371d7f13b834b088a3e199b2ad9c1da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:08.721189 2020 raft_consensus.cc:504] T 0495ad8bb97048ca893c91004aa30bae P 6371d7f13b834b088a3e199b2ad9c1da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } } peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } }
I20260619 16:49:08.721602 2020 leader_election.cc:288] T 0495ad8bb97048ca893c91004aa30bae P 6371d7f13b834b088a3e199b2ad9c1da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f265f16f78f459fa0bf39bec82cee7d (127.0.98.1:34185), 4ee5b1c2133f44d0b27c33bebfb6ad4b (127.0.98.3:37749)
I20260619 16:49:08.725984 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.722153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45319 (local address 127.0.98.3:37749)
0619 16:49:08.722229 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:08.722231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:08.722265 (+ 34us) server_negotiation.cc:407] Connection header received
0619 16:49:08.722397 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:08.722399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:08.722410 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:08.722493 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:08.722636 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.723190 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.724882 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.725472 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.725561 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:08.725577 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:08.725832 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 16:49:08.725914 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 16:49:08.725984 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.721938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58059 (local address 127.0.98.1:34185)
0619 16:49:08.722002 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:08.722004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:08.722089 (+ 85us) server_negotiation.cc:407] Connection header received
0619 16:49:08.722315 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:08.722317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:08.722335 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:08.722396 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:08.722629 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.723189 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.724826 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.725478 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.725561 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:08.725577 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:08.725828 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 16:49:08.725897 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 16:49:08.726037 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.721843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34185 (local address 127.0.98.2:58059)
0619 16:49:08.721998 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:08.722012 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:08.722096 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:08.722432 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:08.722433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:08.722456 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:08.722555 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:08.722558 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.723302 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.723302 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:08.724704 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:08.724706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.725605 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.725606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:08.725716 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:08.725722 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 16:49:08.725791 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 16:49:08.725857 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":39,"thread_start_us":76,"threads_started":1}
I20260619 16:49:08.726037 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.722039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37749 (local address 127.0.98.2:45319)
0619 16:49:08.722162 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:08.722174 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:08.722284 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:08.722504 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:08.722505 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:08.722506 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:08.722567 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:08.722568 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.723270 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.723271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:08.724698 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:08.724703 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.725604 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.725606 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:08.725716 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:08.725724 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 16:49:08.725791 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 16:49:08.725857 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":120,"thread_start_us":42,"threads_started":1}
I20260619 16:49:08.726336 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0495ad8bb97048ca893c91004aa30bae" candidate_uuid: "6371d7f13b834b088a3e199b2ad9c1da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" is_pre_election: true
I20260619 16:49:08.726387 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0495ad8bb97048ca893c91004aa30bae" candidate_uuid: "6371d7f13b834b088a3e199b2ad9c1da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f265f16f78f459fa0bf39bec82cee7d" is_pre_election: true
I20260619 16:49:08.726462 1944 raft_consensus.cc:2454] T 0495ad8bb97048ca893c91004aa30bae P 4ee5b1c2133f44d0b27c33bebfb6ad4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6371d7f13b834b088a3e199b2ad9c1da in term 0.
I20260619 16:49:08.726485 1664 raft_consensus.cc:2454] T 0495ad8bb97048ca893c91004aa30bae P 8f265f16f78f459fa0bf39bec82cee7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6371d7f13b834b088a3e199b2ad9c1da in term 0.
I20260619 16:49:08.726681 1736 leader_election.cc:302] T 0495ad8bb97048ca893c91004aa30bae P 6371d7f13b834b088a3e199b2ad9c1da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ee5b1c2133f44d0b27c33bebfb6ad4b, 6371d7f13b834b088a3e199b2ad9c1da; no voters:
I20260619 16:49:08.726852 2020 raft_consensus.cc:2781] T 0495ad8bb97048ca893c91004aa30bae P 6371d7f13b834b088a3e199b2ad9c1da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:49:08.726910 2020 raft_consensus.cc:482] T 0495ad8bb97048ca893c91004aa30bae P 6371d7f13b834b088a3e199b2ad9c1da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:49:08.726925 2020 raft_consensus.cc:3037] T 0495ad8bb97048ca893c91004aa30bae P 6371d7f13b834b088a3e199b2ad9c1da [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:08.727893 2020 raft_consensus.cc:504] T 0495ad8bb97048ca893c91004aa30bae P 6371d7f13b834b088a3e199b2ad9c1da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } } peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } }
I20260619 16:49:08.728052 2020 leader_election.cc:288] T 0495ad8bb97048ca893c91004aa30bae P 6371d7f13b834b088a3e199b2ad9c1da [CANDIDATE]: Term 1 election: Requested vote from peers 8f265f16f78f459fa0bf39bec82cee7d (127.0.98.1:34185), 4ee5b1c2133f44d0b27c33bebfb6ad4b (127.0.98.3:37749)
I20260619 16:49:08.728267 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0495ad8bb97048ca893c91004aa30bae" candidate_uuid: "6371d7f13b834b088a3e199b2ad9c1da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f265f16f78f459fa0bf39bec82cee7d"
I20260619 16:49:08.728363 1664 raft_consensus.cc:3037] T 0495ad8bb97048ca893c91004aa30bae P 8f265f16f78f459fa0bf39bec82cee7d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:08.728295 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0495ad8bb97048ca893c91004aa30bae" candidate_uuid: "6371d7f13b834b088a3e199b2ad9c1da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b"
I20260619 16:49:08.728466 1944 raft_consensus.cc:3037] T 0495ad8bb97048ca893c91004aa30bae P 4ee5b1c2133f44d0b27c33bebfb6ad4b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:08.729151 1664 raft_consensus.cc:2454] T 0495ad8bb97048ca893c91004aa30bae P 8f265f16f78f459fa0bf39bec82cee7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6371d7f13b834b088a3e199b2ad9c1da in term 1.
I20260619 16:49:08.729152 1944 raft_consensus.cc:2454] T 0495ad8bb97048ca893c91004aa30bae P 4ee5b1c2133f44d0b27c33bebfb6ad4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6371d7f13b834b088a3e199b2ad9c1da in term 1.
I20260619 16:49:08.729403 1736 leader_election.cc:302] T 0495ad8bb97048ca893c91004aa30bae P 6371d7f13b834b088a3e199b2ad9c1da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6371d7f13b834b088a3e199b2ad9c1da, 8f265f16f78f459fa0bf39bec82cee7d; no voters:
I20260619 16:49:08.729548 2020 raft_consensus.cc:2781] T 0495ad8bb97048ca893c91004aa30bae P 6371d7f13b834b088a3e199b2ad9c1da [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:08.729698 2020 raft_consensus.cc:686] T 0495ad8bb97048ca893c91004aa30bae P 6371d7f13b834b088a3e199b2ad9c1da [term 1 LEADER]: Becoming Leader. State: Replica: 6371d7f13b834b088a3e199b2ad9c1da, State: Running, Role: LEADER
I20260619 16:49:08.729781 2020 consensus_queue.cc:237] T 0495ad8bb97048ca893c91004aa30bae P 6371d7f13b834b088a3e199b2ad9c1da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } } peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } }
I20260619 16:49:08.730240 2057 tablet_replica.cc:439] T 0495ad8bb97048ca893c91004aa30bae P 6371d7f13b834b088a3e199b2ad9c1da: TxnStatusTablet state changed. Reason: New leader 6371d7f13b834b088a3e199b2ad9c1da. Latest consensus state: current_term: 1 leader_uuid: "6371d7f13b834b088a3e199b2ad9c1da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } } peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } } }
I20260619 16:49:08.730351 2057 tablet_replica.cc:442] T 0495ad8bb97048ca893c91004aa30bae P 6371d7f13b834b088a3e199b2ad9c1da: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:49:08.730662 2059 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 16:49:08.730785 2059 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 16:49:08.730957 1506 catalog_manager.cc:5495] T 0495ad8bb97048ca893c91004aa30bae P 6371d7f13b834b088a3e199b2ad9c1da reported cstate change: term changed from 0 to 1, leader changed from <none> to 6371d7f13b834b088a3e199b2ad9c1da (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6371d7f13b834b088a3e199b2ad9c1da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } health_report { overall_health: UNKNOWN } } }
I20260619 16:49:08.738561 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.735272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50048 (local address 127.0.98.2:36867)
0619 16:49:08.735335 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:08.735338 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:08.735347 (+ 9us) server_negotiation.cc:407] Connection header received
0619 16:49:08.735414 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:08.735415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:08.735434 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:08.735497 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:08.735723 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.736484 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.737476 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.737685 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.737743 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:08.737955 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:08.738295 (+ 340us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:08.738380 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:08.738436 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 16:49:08.738463 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 16:49:08.738584 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.735142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36867 (local address 127.0.0.1:50048)
0619 16:49:08.735259 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:08.735273 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:08.735327 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:08.735524 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:08.735525 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:08.735527 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:49:08.735600 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:08.735604 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.736573 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.736575 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:08.737376 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:08.737381 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.737761 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.737763 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:08.737832 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:08.737856 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:49:08.738370 (+ 514us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:49:08.738385 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 16:49:08.738432 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 16:49:08.738506 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 16:49:08.740154 1944 raft_consensus.cc:1261] T 0495ad8bb97048ca893c91004aa30bae P 4ee5b1c2133f44d0b27c33bebfb6ad4b [term 1 FOLLOWER]: Refusing update from remote peer 6371d7f13b834b088a3e199b2ad9c1da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:08.740476 1664 raft_consensus.cc:1261] T 0495ad8bb97048ca893c91004aa30bae P 8f265f16f78f459fa0bf39bec82cee7d [term 1 FOLLOWER]: Refusing update from remote peer 6371d7f13b834b088a3e199b2ad9c1da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:08.740490 2057 consensus_queue.cc:1035] T 0495ad8bb97048ca893c91004aa30bae P 6371d7f13b834b088a3e199b2ad9c1da [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:08.740764 2057 consensus_queue.cc:1035] T 0495ad8bb97048ca893c91004aa30bae P 6371d7f13b834b088a3e199b2ad9c1da [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:08.741211 2015 tablet_replica.cc:439] T 0495ad8bb97048ca893c91004aa30bae P 4ee5b1c2133f44d0b27c33bebfb6ad4b: TxnStatusTablet state changed. Reason: New leader 6371d7f13b834b088a3e199b2ad9c1da. Latest consensus state: current_term: 1 leader_uuid: "6371d7f13b834b088a3e199b2ad9c1da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } } peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } } }
I20260619 16:49:08.741343 2015 tablet_replica.cc:442] T 0495ad8bb97048ca893c91004aa30bae P 4ee5b1c2133f44d0b27c33bebfb6ad4b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:08.741659 2034 tablet_replica.cc:439] T 0495ad8bb97048ca893c91004aa30bae P 8f265f16f78f459fa0bf39bec82cee7d: TxnStatusTablet state changed. Reason: New leader 6371d7f13b834b088a3e199b2ad9c1da. Latest consensus state: current_term: 1 leader_uuid: "6371d7f13b834b088a3e199b2ad9c1da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } } peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } } }
I20260619 16:49:08.741775 2034 tablet_replica.cc:442] T 0495ad8bb97048ca893c91004aa30bae P 8f265f16f78f459fa0bf39bec82cee7d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:08.742815 2020 tablet_replica.cc:439] T 0495ad8bb97048ca893c91004aa30bae P 6371d7f13b834b088a3e199b2ad9c1da: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6371d7f13b834b088a3e199b2ad9c1da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } } peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } } }
I20260619 16:49:08.743112 2020 tablet_replica.cc:442] T 0495ad8bb97048ca893c91004aa30bae P 6371d7f13b834b088a3e199b2ad9c1da: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:49:08.744280 2015 tablet_replica.cc:439] T 0495ad8bb97048ca893c91004aa30bae P 4ee5b1c2133f44d0b27c33bebfb6ad4b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6371d7f13b834b088a3e199b2ad9c1da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } } peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } } }
I20260619 16:49:08.744405 2015 tablet_replica.cc:442] T 0495ad8bb97048ca893c91004aa30bae P 4ee5b1c2133f44d0b27c33bebfb6ad4b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:08.746562 2020 tablet_replica.cc:439] T 0495ad8bb97048ca893c91004aa30bae P 6371d7f13b834b088a3e199b2ad9c1da: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6371d7f13b834b088a3e199b2ad9c1da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } } peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } } }
I20260619 16:49:08.746716 2020 tablet_replica.cc:442] T 0495ad8bb97048ca893c91004aa30bae P 6371d7f13b834b088a3e199b2ad9c1da: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:49:08.747340 2026 mvcc.cc:205] Tried to move back new op lower bound from 7298612218837307392 to 7298612218797940736. Current Snapshot: MvccSnapshot[applied={T|T < 7298612218837307392}]
I20260619 16:49:08.747469 2034 tablet_replica.cc:439] T 0495ad8bb97048ca893c91004aa30bae P 8f265f16f78f459fa0bf39bec82cee7d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6371d7f13b834b088a3e199b2ad9c1da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 } } peers { permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 } } peers { permanent_uuid: "4ee5b1c2133f44d0b27c33bebfb6ad4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37749 } } }
I20260619 16:49:08.747560 2034 tablet_replica.cc:442] T 0495ad8bb97048ca893c91004aa30bae P 8f265f16f78f459fa0bf39bec82cee7d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:08.750418 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.746569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59674 (local address 127.0.98.3:37749)
0619 16:49:08.746621 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:08.746623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:08.746632 (+ 9us) server_negotiation.cc:407] Connection header received
0619 16:49:08.746681 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:08.746681 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:08.746707 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:08.746807 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:08.747015 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.747718 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.749027 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.749486 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.749565 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:08.749935 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:08.750083 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:08.750166 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:08.750254 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 16:49:08.750306 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 16:49:08.759690 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.754675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36867 (local address 127.0.98.3:50067)
0619 16:49:08.754891 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:08.754903 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:08.754998 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:08.755449 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:08.755450 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:08.755453 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:49:08.755536 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:08.755539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.756806 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.756807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:08.757780 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:08.757786 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.758669 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.758671 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:08.758740 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:08.758767 (+ 27us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:49:08.759225 (+ 458us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:49:08.759229 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:49:08.759290 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 16:49:08.759522 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":112,"threads_started":1}
I20260619 16:49:08.759640 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.754776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50067 (local address 127.0.98.2:36867)
0619 16:49:08.754836 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:08.754839 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:08.754995 (+ 156us) server_negotiation.cc:407] Connection header received
0619 16:49:08.755058 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:08.755059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:08.755071 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:08.755413 (+ 342us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:08.755685 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.756681 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.758065 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.758566 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.758651 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:08.758872 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:08.758979 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:08.759201 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:08.759300 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 16:49:08.759331 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 16:49:08.763316 1804 raft_consensus.cc:1261] T f9a5c735de6f4b54b1fe33c939fb6413 P 6371d7f13b834b088a3e199b2ad9c1da [term 1 FOLLOWER]: Refusing update from remote peer 4ee5b1c2133f44d0b27c33bebfb6ad4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:08.763309 1664 raft_consensus.cc:1261] T f9a5c735de6f4b54b1fe33c939fb6413 P 8f265f16f78f459fa0bf39bec82cee7d [term 1 FOLLOWER]: Refusing update from remote peer 4ee5b1c2133f44d0b27c33bebfb6ad4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:08.763660 2015 consensus_queue.cc:1035] T f9a5c735de6f4b54b1fe33c939fb6413 P 4ee5b1c2133f44d0b27c33bebfb6ad4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6371d7f13b834b088a3e199b2ad9c1da" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:08.763849 2044 consensus_queue.cc:1035] T f9a5c735de6f4b54b1fe33c939fb6413 P 4ee5b1c2133f44d0b27c33bebfb6ad4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f265f16f78f459fa0bf39bec82cee7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:08.775655 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.771881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60409 (local address 127.0.98.3:37749)
0619 16:49:08.771949 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:08.771952 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:08.772120 (+ 168us) server_negotiation.cc:407] Connection header received
0619 16:49:08.772177 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:08.772178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:08.772190 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:08.772244 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:08.772405 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.773059 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.774710 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.774961 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.775055 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:08.775319 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:08.775362 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:08.775425 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:08.775507 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 16:49:08.775548 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 16:49:08.775689 2072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.771791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37749 (local address 127.0.98.2:60409)
0619 16:49:08.772031 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:08.772042 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:08.772116 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:08.772249 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:08.772250 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:08.772252 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:49:08.772345 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:08.772348 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.773176 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.773178 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:08.774616 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:08.774622 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.775057 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.775058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:08.775139 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:08.775169 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:49:08.775438 (+ 269us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:49:08.775440 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 16:49:08.775484 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 16:49:08.775556 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":103,"threads_started":1}
I20260619 16:49:08.784898 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:08.785064 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:08.785859 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7ff670f5ffd941f086bf2f9c2d9955d1: Sending DeleteTablet for 3 replicas of tablet f9a5c735de6f4b54b1fe33c939fb6413
I20260619 16:49:08.786168 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet f9a5c735de6f4b54b1fe33c939fb6413 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50036
I20260619 16:49:08.786326 2039 tablet_replica.cc:330] T f9a5c735de6f4b54b1fe33c939fb6413 P 6371d7f13b834b088a3e199b2ad9c1da: stopping tablet replica
I20260619 16:49:08.786360 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet f9a5c735de6f4b54b1fe33c939fb6413 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35848
I20260619 16:49:08.786410 2039 raft_consensus.cc:2229] T f9a5c735de6f4b54b1fe33c939fb6413 P 6371d7f13b834b088a3e199b2ad9c1da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:08.786453 2039 raft_consensus.cc:2258] T f9a5c735de6f4b54b1fe33c939fb6413 P 6371d7f13b834b088a3e199b2ad9c1da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:08.786473 2037 tablet_replica.cc:330] T f9a5c735de6f4b54b1fe33c939fb6413 P 8f265f16f78f459fa0bf39bec82cee7d: stopping tablet replica
I20260619 16:49:08.786473 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet f9a5c735de6f4b54b1fe33c939fb6413 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59666
I20260619 16:49:08.786553 2037 raft_consensus.cc:2229] T f9a5c735de6f4b54b1fe33c939fb6413 P 8f265f16f78f459fa0bf39bec82cee7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:08.786561 2038 tablet_replica.cc:330] T f9a5c735de6f4b54b1fe33c939fb6413 P 4ee5b1c2133f44d0b27c33bebfb6ad4b: stopping tablet replica
I20260619 16:49:08.786604 2037 raft_consensus.cc:2258] T f9a5c735de6f4b54b1fe33c939fb6413 P 8f265f16f78f459fa0bf39bec82cee7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:08.786610 2038 raft_consensus.cc:2229] T f9a5c735de6f4b54b1fe33c939fb6413 P 4ee5b1c2133f44d0b27c33bebfb6ad4b [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:49:08.786722 2038 raft_consensus.cc:2258] T f9a5c735de6f4b54b1fe33c939fb6413 P 4ee5b1c2133f44d0b27c33bebfb6ad4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:08.787005 2039 ts_tablet_manager.cc:1905] T f9a5c735de6f4b54b1fe33c939fb6413 P 6371d7f13b834b088a3e199b2ad9c1da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:08.787127 2038 ts_tablet_manager.cc:1905] T f9a5c735de6f4b54b1fe33c939fb6413 P 4ee5b1c2133f44d0b27c33bebfb6ad4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:08.789109 2037 ts_tablet_manager.cc:1905] T f9a5c735de6f4b54b1fe33c939fb6413 P 8f265f16f78f459fa0bf39bec82cee7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:08.791280 2037 ts_tablet_manager.cc:1918] T f9a5c735de6f4b54b1fe33c939fb6413 P 8f265f16f78f459fa0bf39bec82cee7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:49:08.791416 2037 log.cc:1196] T f9a5c735de6f4b54b1fe33c939fb6413 P 8f265f16f78f459fa0bf39bec82cee7d: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-0/wal/wals/f9a5c735de6f4b54b1fe33c939fb6413
I20260619 16:49:08.791661 2037 ts_tablet_manager.cc:1939] T f9a5c735de6f4b54b1fe33c939fb6413 P 8f265f16f78f459fa0bf39bec82cee7d: Deleting consensus metadata
I20260619 16:49:08.792243 1492 catalog_manager.cc:4841] TS 8f265f16f78f459fa0bf39bec82cee7d (127.0.98.1:34185): tablet f9a5c735de6f4b54b1fe33c939fb6413 (table test-table [id=9e30ecd4082e46918aa61191d7fe0d8b]) successfully deleted
I20260619 16:49:08.792451 2038 ts_tablet_manager.cc:1918] T f9a5c735de6f4b54b1fe33c939fb6413 P 4ee5b1c2133f44d0b27c33bebfb6ad4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:49:08.792524 2038 log.cc:1196] T f9a5c735de6f4b54b1fe33c939fb6413 P 4ee5b1c2133f44d0b27c33bebfb6ad4b: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-2/wal/wals/f9a5c735de6f4b54b1fe33c939fb6413
I20260619 16:49:08.792863 2038 ts_tablet_manager.cc:1939] T f9a5c735de6f4b54b1fe33c939fb6413 P 4ee5b1c2133f44d0b27c33bebfb6ad4b: Deleting consensus metadata
I20260619 16:49:08.793375 2039 ts_tablet_manager.cc:1918] T f9a5c735de6f4b54b1fe33c939fb6413 P 6371d7f13b834b088a3e199b2ad9c1da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:49:08.793447 2039 log.cc:1196] T f9a5c735de6f4b54b1fe33c939fb6413 P 6371d7f13b834b088a3e199b2ad9c1da: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781887741903642-392-0/minicluster-data/ts-1/wal/wals/f9a5c735de6f4b54b1fe33c939fb6413
I20260619 16:49:08.793644 2039 ts_tablet_manager.cc:1939] T f9a5c735de6f4b54b1fe33c939fb6413 P 6371d7f13b834b088a3e199b2ad9c1da: Deleting consensus metadata
I20260619 16:49:08.793946 1492 catalog_manager.cc:4841] TS 4ee5b1c2133f44d0b27c33bebfb6ad4b (127.0.98.3:37749): tablet f9a5c735de6f4b54b1fe33c939fb6413 (table test-table [id=9e30ecd4082e46918aa61191d7fe0d8b]) successfully deleted
I20260619 16:49:08.794008 1491 catalog_manager.cc:4841] TS 6371d7f13b834b088a3e199b2ad9c1da (127.0.98.2:36867): tablet f9a5c735de6f4b54b1fe33c939fb6413 (table test-table [id=9e30ecd4082e46918aa61191d7fe0d8b]) successfully deleted
Jun 19 16:49:08 dist-test-slave-jsw9 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887748, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 16:49:08.805653 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.792078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35876 (local address 127.0.98.1:34185)
0619 16:49:08.792169 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:08.792171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:08.794222 (+ 2051us) server_negotiation.cc:407] Connection header received
0619 16:49:08.794322 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:08.794323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:08.794338 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:08.794415 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:08.795315 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.796170 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.797372 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.797701 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.797775 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:08.801826 (+ 4051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:08.801844 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:08.801844 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:08.801856 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:08.804129 (+ 2273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:08.804666 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:08.804671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:08.804672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:08.804765 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:08.804983 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:08.804984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:08.804985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:08.805197 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:08.805361 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:08.805429 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 16:49:08.805532 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260619 16:49:08.806455 1704 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35876
I20260619 16:49:08.817179 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:08.807653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51004 (local address 127.0.98.62:38001)
0619 16:49:08.807706 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:08.807708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:08.807933 (+ 225us) server_negotiation.cc:407] Connection header received
0619 16:49:08.807991 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:08.807993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:08.808007 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:08.808065 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:08.808782 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.809668 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.810974 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:08.811370 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:08.811450 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:08.813016 (+ 1566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:08.813018 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:08.813019 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:08.813028 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:08.815074 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:08.815951 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:08.815953 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:08.815953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:08.816044 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:08.816287 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:08.816288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:08.816289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:08.816469 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:08.816591 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:08.816945 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 16:49:08.817060 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260619 16:49:08.817950 1507 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51004
I20260619 16:49:08.819357 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 1579
I20260619 16:49:08.828969 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 1719
I20260619 16:49:08.837059 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 1859
I20260619 16:49:08.842669 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 1475
2026-06-19T16:49:08Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2983 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:49:08 dist-test-slave-jsw9 krb5kdc[2081](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:49:08 dist-test-slave-jsw9 krb5kdc[2081](info): set up 2 sockets
Jun 19 16:49:08 dist-test-slave-jsw9 krb5kdc[2081](info): commencing operation
krb5kdc: starting...
W20260619 16:49:09.842175 392 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:49:09 dist-test-slave-jsw9 krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:49:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:49:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:49:09.982947 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46337
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:10.082238 2094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:10.082572 2094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:10.082603 2094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:10.084005 2094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:10.084048 2094 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 16:49:10.084059 2094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:10.084071 2094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:10.084084 2094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:10.084093 2094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:49:10.085741 2094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46337
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:10.086002 2094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:10.086274 2094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:10.091998 2099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:10.092099 2102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:10.092087 2100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:10.092089 2094 server_base.cc:1034] running on GCE node
I20260619 16:49:10.092600 2094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:10.092963 2094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:10.094157 2094 hybrid_clock.cc:648] HybridClock initialized: now 1781887750094117 us; error 56 us; skew 500 ppm
Jun 19 16:49:10 dist-test-slave-jsw9 krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:10.097506 2094 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:49:10.098309 2094 webserver.cc:460] Webserver started at http://127.0.98.62:38607/ using document root <none> and password file <none>
I20260619 16:49:10.098618 2094 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:10.098788 2094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:10.098955 2094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:10.100253 2094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "8990279310d642648682b1df509b9968"
format_stamp: "Formatted at 2026-06-19 16:49:10 on dist-test-slave-jsw9"
server_key: "2b993980a32b6db41d05ec636862d5a1"
server_key_iv: "83b0b7dc775c8eee89cc03b6a5b4c3ed"
server_key_version: "encryptionkey@0"
I20260619 16:49:10.100653 2094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "8990279310d642648682b1df509b9968"
format_stamp: "Formatted at 2026-06-19 16:49:10 on dist-test-slave-jsw9"
server_key: "2b993980a32b6db41d05ec636862d5a1"
server_key_iv: "83b0b7dc775c8eee89cc03b6a5b4c3ed"
server_key_version: "encryptionkey@0"
I20260619 16:49:10.102767 2094 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260619 16:49:10.103758 2109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:10.104138 2094 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:49:10.104259 2094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "8990279310d642648682b1df509b9968"
format_stamp: "Formatted at 2026-06-19 16:49:10 on dist-test-slave-jsw9"
server_key: "2b993980a32b6db41d05ec636862d5a1"
server_key_iv: "83b0b7dc775c8eee89cc03b6a5b4c3ed"
server_key_version: "encryptionkey@0"
I20260619 16:49:10.104359 2094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:10.125286 2094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:10.127825 2094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:10.127997 2094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:10.133502 2094 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46337
I20260619 16:49:10.133553 2171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46337 every 8 connection(s)
I20260619 16:49:10.133960 2094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:10.134769 2172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:10.137579 2172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8990279310d642648682b1df509b9968: Bootstrap starting.
I20260619 16:49:10.138393 2172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8990279310d642648682b1df509b9968: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:10.138680 2172 log.cc:824] T 00000000000000000000000000000000 P 8990279310d642648682b1df509b9968: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:10.139514 2172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8990279310d642648682b1df509b9968: No bootstrap required, opened a new log
I20260619 16:49:10.140065 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 2094
I20260619 16:49:10.140156 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:49:10.140453 392 external_mini_cluster.cc:1428] Setting key 01b313aa8901479e372fc6494248ff8b
I20260619 16:49:10.142369 2172 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8990279310d642648682b1df509b9968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8990279310d642648682b1df509b9968" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46337 } }
I20260619 16:49:10.142589 2172 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8990279310d642648682b1df509b9968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:10.142629 2172 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8990279310d642648682b1df509b9968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8990279310d642648682b1df509b9968, State: Initialized, Role: FOLLOWER
I20260619 16:49:10.142793 2172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8990279310d642648682b1df509b9968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8990279310d642648682b1df509b9968" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46337 } }
I20260619 16:49:10.142897 2172 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8990279310d642648682b1df509b9968 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:10.142941 2172 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8990279310d642648682b1df509b9968 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:10.142997 2172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8990279310d642648682b1df509b9968 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:10.143862 2172 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8990279310d642648682b1df509b9968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8990279310d642648682b1df509b9968" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46337 } }
I20260619 16:49:10.143981 2172 leader_election.cc:302] T 00000000000000000000000000000000 P 8990279310d642648682b1df509b9968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8990279310d642648682b1df509b9968; no voters:
I20260619 16:49:10.144276 2172 leader_election.cc:288] T 00000000000000000000000000000000 P 8990279310d642648682b1df509b9968 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:49:10.144362 2177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8990279310d642648682b1df509b9968 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:10.144488 2177 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8990279310d642648682b1df509b9968 [term 1 LEADER]: Becoming Leader. State: Replica: 8990279310d642648682b1df509b9968, State: Running, Role: LEADER
I20260619 16:49:10.144580 2177 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8990279310d642648682b1df509b9968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8990279310d642648682b1df509b9968" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46337 } }
I20260619 16:49:10.144647 2172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8990279310d642648682b1df509b9968 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:10.144976 2177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8990279310d642648682b1df509b9968 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8990279310d642648682b1df509b9968" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8990279310d642648682b1df509b9968" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46337 } } }
I20260619 16:49:10.145056 2177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8990279310d642648682b1df509b9968 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:10.145170 2177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8990279310d642648682b1df509b9968 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8990279310d642648682b1df509b9968. Latest consensus state: current_term: 1 leader_uuid: "8990279310d642648682b1df509b9968" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8990279310d642648682b1df509b9968" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46337 } } }
I20260619 16:49:10.145231 2177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8990279310d642648682b1df509b9968 [sys.catalog]: This master's current role is: LEADER
W20260619 16:49:10.146745 2193 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8990279310d642648682b1df509b9968: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:49:10.146817 2193 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:49:10.146870 2182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:10.147064 2182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:10.148883 2182 catalog_manager.cc:1347] Generated new cluster ID: 5dff2f07a1e04280aa743f5208b4e2eb
I20260619 16:49:10.148941 2182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 16:49:10 dist-test-slave-jsw9 krb5kdc[2081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:10.153244 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:10.141341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35916 (local address 127.0.98.62:46337)
0619 16:49:10.141536 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:10.141546 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:10.141580 (+ 34us) server_negotiation.cc:407] Connection header received
0619 16:49:10.142313 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:10.142326 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:10.142401 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:10.142571 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:10.143429 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.144809 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.145838 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.146587 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.146714 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:10.149589 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:10.149593 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:10.149594 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:10.149607 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:10.151494 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:10.151878 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:10.151881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:10.151881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:10.151998 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:10.152154 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:10.152155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:10.152155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:10.152474 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:10.152624 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:10.152838 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 16:49:10.153005 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":87,"threads_started":1}
I20260619 16:49:10.161302 2182 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:49:10.162154 2182 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:10.169239 2182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8990279310d642648682b1df509b9968: Generated new TSK 0
I20260619 16:49:10.169500 2182 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:49:10.201094 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46337
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:10.305052 2198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:10.305356 2198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:10.305392 2198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:10.307008 2198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:10.307073 2198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:10.307098 2198 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 16:49:10.307109 2198 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:49:10.307128 2198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:49:10.308979 2198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:10.309316 2198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:10.309612 2198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:10.315438 2204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:10.315440 2203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:10.315709 2198 server_base.cc:1034] running on GCE node
W20260619 16:49:10.315515 2206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:10.315999 2198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:10.316287 2198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:10.317449 2198 hybrid_clock.cc:648] HybridClock initialized: now 1781887750317437 us; error 37 us; skew 500 ppm
Jun 19 16:49:10 dist-test-slave-jsw9 krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:10.321009 2198 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:49:10.321542 2198 webserver.cc:460] Webserver started at http://127.0.98.1:36551/ using document root <none> and password file <none>
I20260619 16:49:10.321792 2198 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:10.321848 2198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:10.321973 2198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:10.323236 2198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-0/data/instance:
uuid: "bc51e846ab01481e823acbcb5b5a9211"
format_stamp: "Formatted at 2026-06-19 16:49:10 on dist-test-slave-jsw9"
server_key: "dde8798241856b899ce308f2727fce4c"
server_key_iv: "7c1192dce58dd18aaa7273d809e424d1"
server_key_version: "encryptionkey@0"
I20260619 16:49:10.323621 2198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bc51e846ab01481e823acbcb5b5a9211"
format_stamp: "Formatted at 2026-06-19 16:49:10 on dist-test-slave-jsw9"
server_key: "dde8798241856b899ce308f2727fce4c"
server_key_iv: "7c1192dce58dd18aaa7273d809e424d1"
server_key_version: "encryptionkey@0"
I20260619 16:49:10.325073 2198 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 16:49:10.325968 2213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:10.326277 2198 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 16:49:10.326421 2198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-0/wal
uuid: "bc51e846ab01481e823acbcb5b5a9211"
format_stamp: "Formatted at 2026-06-19 16:49:10 on dist-test-slave-jsw9"
server_key: "dde8798241856b899ce308f2727fce4c"
server_key_iv: "7c1192dce58dd18aaa7273d809e424d1"
server_key_version: "encryptionkey@0"
I20260619 16:49:10.326512 2198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:10.347502 2198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:10.350102 2198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:10.350394 2198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:10.351061 2198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:10.351112 2198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:10.351146 2198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:10.351163 2198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:49:10 dist-test-slave-jsw9 krb5kdc[2081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:10.361363 2198 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39321
I20260619 16:49:10.361909 2198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:49:10.362313 2328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39321 every 8 connection(s)
I20260619 16:49:10.364840 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:10.351620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36415 (local address 127.0.98.62:46337)
0619 16:49:10.351675 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:10.351677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:10.351982 (+ 305us) server_negotiation.cc:407] Connection header received
0619 16:49:10.352720 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:10.352722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:10.352746 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:10.352779 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:10.353917 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.355746 (+ 1829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.357026 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.357412 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.357505 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:10.360241 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:10.360244 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:10.360244 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:10.360263 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:10.362931 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:10.363508 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:10.363510 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:10.363511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:10.363602 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:10.363946 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:10.363947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:10.363948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:10.364121 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:10.364197 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:10.364637 (+ 440us) server_negotiation.cc:299] Negotiation successful
0619 16:49:10.364762 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 16:49:10.364982 2222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:10.351626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46337 (local address 127.0.98.1:36415)
0619 16:49:10.351819 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:10.351860 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:10.352575 (+ 715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:10.353013 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:10.353019 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:10.353420 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:10.353779 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:10.353788 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.355920 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.355922 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:10.356940 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:10.356944 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.357534 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.357535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:10.357709 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:10.358462 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:10.358473 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:10.360118 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:10.363083 (+ 2965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:10.363090 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:10.363093 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:10.363419 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:10.363665 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:10.363665 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:10.363666 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:10.363830 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:10.364208 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:10.364214 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:10.364429 (+ 215us) client_negotiation.cc:769] Sending connection context
0619 16:49:10.364535 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 16:49:10.364739 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":113,"threads_started":1}
I20260619 16:49:10.365639 2329 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46337
I20260619 16:49:10.365800 2329 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:10.366063 2329 heartbeater.cc:511] Master 127.0.98.62:46337 requested a full tablet report, sending...
I20260619 16:49:10.366782 2125 ts_manager.cc:194] Registered new tserver with Master: bc51e846ab01481e823acbcb5b5a9211 (127.0.98.1:39321)
I20260619 16:49:10.367712 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 2198
I20260619 16:49:10.367722 2125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36415
I20260619 16:49:10.367822 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:49:10.367997 392 external_mini_cluster.cc:1428] Setting key f7c253a86baf41a3b6c922d85855e466
I20260619 16:49:10.374532 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:10.367062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51845 (local address 127.0.98.62:46337)
0619 16:49:10.367114 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:10.367116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:10.367221 (+ 105us) server_negotiation.cc:407] Connection header received
0619 16:49:10.367286 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:10.367287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:10.367299 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:10.367352 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:10.367776 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.368769 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.369772 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.369995 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.370066 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:10.371314 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:10.371315 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:10.371315 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:10.371326 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:10.372972 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:10.373402 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:10.373404 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:10.373404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:10.373489 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:10.373687 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:10.373688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:10.373689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:10.373883 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:10.373967 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:10.374260 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 16:49:10.374396 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":183,"server-negotiator.queue_time_us":23}
I20260619 16:49:10.374557 2335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:10.366986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46337 (local address 127.0.98.1:51845)
0619 16:49:10.367150 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:10.367162 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:10.367227 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:10.367357 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:10.367358 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:10.367581 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:10.367682 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:10.367686 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.368884 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.368886 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:10.369663 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:10.369668 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.370073 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.370074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:10.370160 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:10.370762 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:10.370769 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:10.371173 (+ 404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:10.373076 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:10.373077 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:10.373077 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:10.373287 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:10.373563 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:10.373563 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:10.373564 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:10.373630 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:10.373984 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:10.373986 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:10.374058 (+ 72us) client_negotiation.cc:769] Sending connection context
0619 16:49:10.374137 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 16:49:10.374453 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:49:10.398361 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46337
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:10.494376 2338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:10.494659 2338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:10.494693 2338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:10.496301 2338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:10.496428 2338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:10.496467 2338 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 16:49:10.496483 2338 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:49:10.496510 2338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:49:10.498721 2338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:10.499101 2338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:10.499377 2338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:49:10.506304 2338 server_base.cc:1034] running on GCE node
W20260619 16:49:10.506171 2344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:10.506170 2343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:10.506170 2346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:10.506712 2338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:10.507006 2338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:10.508239 2338 hybrid_clock.cc:648] HybridClock initialized: now 1781887750508187 us; error 67 us; skew 500 ppm
Jun 19 16:49:10 dist-test-slave-jsw9 krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:10.511653 2338 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:49:10.512377 2338 webserver.cc:460] Webserver started at http://127.0.98.2:40251/ using document root <none> and password file <none>
I20260619 16:49:10.512674 2338 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:10.512735 2338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:10.512876 2338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:10.514168 2338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-1/data/instance:
uuid: "2bca37447f014aeca45e6117d296bc8e"
format_stamp: "Formatted at 2026-06-19 16:49:10 on dist-test-slave-jsw9"
server_key: "51ab2c059b3d41c350778147f9924d5e"
server_key_iv: "83b4d573ef3fc3e26311459675e5e4be"
server_key_version: "encryptionkey@0"
I20260619 16:49:10.514631 2338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2bca37447f014aeca45e6117d296bc8e"
format_stamp: "Formatted at 2026-06-19 16:49:10 on dist-test-slave-jsw9"
server_key: "51ab2c059b3d41c350778147f9924d5e"
server_key_iv: "83b4d573ef3fc3e26311459675e5e4be"
server_key_version: "encryptionkey@0"
I20260619 16:49:10.516178 2338 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 16:49:10.517108 2353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:10.517344 2338 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 16:49:10.517444 2338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-1/wal
uuid: "2bca37447f014aeca45e6117d296bc8e"
format_stamp: "Formatted at 2026-06-19 16:49:10 on dist-test-slave-jsw9"
server_key: "51ab2c059b3d41c350778147f9924d5e"
server_key_iv: "83b4d573ef3fc3e26311459675e5e4be"
server_key_version: "encryptionkey@0"
I20260619 16:49:10.517530 2338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:10.560684 2338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:10.563642 2338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:10.563830 2338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:10.564497 2338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:10.564548 2338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:10.564584 2338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:10.564600 2338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:10.573535 2338 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41853
I20260619 16:49:10.573632 2468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41853 every 8 connection(s)
Jun 19 16:49:10 dist-test-slave-jsw9 krb5kdc[2081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:10.574124 2338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:49:10.575344 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 2338
I20260619 16:49:10.575440 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:49:10.575640 392 external_mini_cluster.cc:1428] Setting key 7b81062fb1176be97a5dab6dd3b86774
I20260619 16:49:10.579811 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:10.565102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48355 (local address 127.0.98.62:46337)
0619 16:49:10.565153 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:10.565155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:10.565570 (+ 415us) server_negotiation.cc:407] Connection header received
0619 16:49:10.566345 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:10.566348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:10.566371 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:10.566465 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:10.567494 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.568348 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.570945 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.571350 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.571443 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:10.575297 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:10.575300 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:10.575301 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:10.575312 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:10.577716 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:10.578580 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:10.578581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:10.578581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:10.578652 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:10.578945 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:10.578945 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:10.578946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:10.579100 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:10.579162 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:10.579597 (+ 435us) server_negotiation.cc:299] Negotiation successful
0619 16:49:10.579702 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 16:49:10.580546 2362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:10.565176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46337 (local address 127.0.98.2:48355)
0619 16:49:10.565428 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:10.565475 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:10.566147 (+ 672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:10.566567 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:10.566573 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:10.566995 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:10.567354 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:10.567362 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.568443 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.568444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:10.570818 (+ 2374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:10.570823 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.571471 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.571472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:10.571643 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:10.572334 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:10.572345 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:10.575158 (+ 2813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:10.577815 (+ 2657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:10.577826 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:10.577829 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:10.578351 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:10.578734 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:10.578735 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:10.578736 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:10.578870 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:10.579163 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:10.579171 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:10.579341 (+ 170us) client_negotiation.cc:769] Sending connection context
0619 16:49:10.579559 (+ 218us) client_negotiation.cc:240] Negotiation successful
0619 16:49:10.579786 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":112,"threads_started":1}
I20260619 16:49:10.583523 2469 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46337
I20260619 16:49:10.584492 2469 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:10.584784 2469 heartbeater.cc:511] Master 127.0.98.62:46337 requested a full tablet report, sending...
I20260619 16:49:10.587430 2126 ts_manager.cc:194] Registered new tserver with Master: 2bca37447f014aeca45e6117d296bc8e (127.0.98.2:41853)
I20260619 16:49:10.588099 2126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48355
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 16:49:10.596902 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:10.586450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55089 (local address 127.0.98.62:46337)
0619 16:49:10.587862 (+ 1412us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:10.587864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:10.587885 (+ 21us) server_negotiation.cc:407] Connection header received
0619 16:49:10.587916 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:10.587917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:10.587930 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:10.587998 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:10.588590 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.590091 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.591137 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.591559 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.591644 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:10.592964 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:10.592966 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:10.592966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:10.592976 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:10.595588 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:10.595996 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:10.595998 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:10.595999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:10.596063 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:10.596236 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:10.596237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:10.596238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:10.596455 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:10.596524 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:10.596677 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 16:49:10.596776 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1348}
I20260619 16:49:10.596917 2476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:10.585316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46337 (local address 127.0.98.2:55089)
0619 16:49:10.585531 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:10.585576 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:10.585674 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:10.588064 (+ 2390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:10.588066 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:10.588408 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:10.588491 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:10.588495 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.590247 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.590249 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:10.590995 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:10.591000 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.591734 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.591735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:10.591827 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:10.592475 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:10.592480 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:10.592848 (+ 368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:10.595711 (+ 2863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:10.595713 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:10.595714 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:10.595922 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:10.596121 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:10.596122 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:10.596122 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:10.596184 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:10.596524 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:10.596525 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:10.596585 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 16:49:10.596645 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 16:49:10.596778 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":113,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:49:10.618706 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46337
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:10.719179 2478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:10.719432 2478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:10.719460 2478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:10.720731 2478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:10.720794 2478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:10.720817 2478 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 16:49:10.720829 2478 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:49:10.720844 2478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:49:10.722954 2478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:10.723485 2478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:10.723757 2478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:10.729684 2483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:10.729684 2478 server_base.cc:1034] running on GCE node
W20260619 16:49:10.729713 2486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:10.729700 2484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:10.730130 2478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:10.730430 2478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:10.731629 2478 hybrid_clock.cc:648] HybridClock initialized: now 1781887750731614 us; error 31 us; skew 500 ppm
Jun 19 16:49:10 dist-test-slave-jsw9 krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:10.734449 2478 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:49:10.735003 2478 webserver.cc:460] Webserver started at http://127.0.98.3:34079/ using document root <none> and password file <none>
I20260619 16:49:10.735242 2478 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:10.735289 2478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:10.735423 2478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:10.736752 2478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-2/data/instance:
uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0"
format_stamp: "Formatted at 2026-06-19 16:49:10 on dist-test-slave-jsw9"
server_key: "96af98ef0ed72b7ca80ef2527a0fe7c6"
server_key_iv: "49b33a893fe0e5e9056acd7d289c47b6"
server_key_version: "encryptionkey@0"
I20260619 16:49:10.737566 2478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0"
format_stamp: "Formatted at 2026-06-19 16:49:10 on dist-test-slave-jsw9"
server_key: "96af98ef0ed72b7ca80ef2527a0fe7c6"
server_key_iv: "49b33a893fe0e5e9056acd7d289c47b6"
server_key_version: "encryptionkey@0"
I20260619 16:49:10.740410 2478 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:49:10.744295 2493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:10.744593 2478 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260619 16:49:10.744725 2478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-2/wal
uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0"
format_stamp: "Formatted at 2026-06-19 16:49:10 on dist-test-slave-jsw9"
server_key: "96af98ef0ed72b7ca80ef2527a0fe7c6"
server_key_iv: "49b33a893fe0e5e9056acd7d289c47b6"
server_key_version: "encryptionkey@0"
I20260619 16:49:10.744903 2478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:10.761605 2478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:10.764808 2478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:10.764993 2478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:10.765797 2478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:10.765862 2478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:10.765897 2478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:10.765913 2478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:49:10 dist-test-slave-jsw9 krb5kdc[2081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:10.774271 2478 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32883
I20260619 16:49:10.774369 2608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32883 every 8 connection(s)
I20260619 16:49:10.774717 2478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:49:10.775281 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 2478
I20260619 16:49:10.775393 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:49:10.775625 392 external_mini_cluster.cc:1428] Setting key bc85b2c524fd01568224d8785025cdec
I20260619 16:49:10.777153 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:10.766533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35183 (local address 127.0.98.62:46337)
0619 16:49:10.766599 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:10.766602 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:10.766981 (+ 379us) server_negotiation.cc:407] Connection header received
0619 16:49:10.767604 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:10.767605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:10.767620 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:10.767675 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:10.768471 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.769024 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.770159 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.770506 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.770566 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:10.773711 (+ 3145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:10.773714 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:10.773714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:10.773725 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:10.775556 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:10.776075 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:10.776076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:10.776077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:10.776149 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:10.776403 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:10.776405 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:10.776406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:10.776576 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:10.776643 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:10.776953 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 16:49:10.777052 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 16:49:10.777436 2502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:10.766594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46337 (local address 127.0.98.3:35183)
0619 16:49:10.766819 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:10.766867 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:10.767485 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:10.767770 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:10.767776 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:10.768102 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:10.768365 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:10.768373 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.769119 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.769120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:10.770059 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:10.770062 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.770595 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.770596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:10.770767 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:10.771419 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:10.771429 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:10.773587 (+ 2158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:10.775667 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:10.775675 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:10.775678 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:10.776002 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:10.776213 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:10.776214 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:10.776214 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:10.776337 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:10.776649 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:10.776656 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:10.776816 (+ 160us) client_negotiation.cc:769] Sending connection context
0619 16:49:10.776924 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 16:49:10.777125 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":105,"threads_started":1}
I20260619 16:49:10.777927 2609 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46337
I20260619 16:49:10.778023 2609 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:10.778262 2609 heartbeater.cc:511] Master 127.0.98.62:46337 requested a full tablet report, sending...
I20260619 16:49:10.778654 2125 ts_manager.cc:194] Registered new tserver with Master: 4a17f838c5e94a6d87248e8ca5ea9ce0 (127.0.98.3:32883)
I20260619 16:49:10.779280 2125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35183
I20260619 16:49:10.786034 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:10.779235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47879 (local address 127.0.98.62:46337)
0619 16:49:10.779284 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:10.779287 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:10.779515 (+ 228us) server_negotiation.cc:407] Connection header received
0619 16:49:10.779590 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:10.779591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:10.779602 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:10.779652 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:10.780026 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.780544 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.781279 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.781478 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.781535 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:10.782901 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:10.782903 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:10.782903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:10.782918 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:10.784767 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:10.785121 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:10.785123 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:10.785123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:10.785181 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:10.785457 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:10.785458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:10.785459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:10.785609 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:10.785673 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:10.785824 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 16:49:10.785919 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 16:49:10.786048 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:10.779231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46337 (local address 127.0.98.3:47879)
0619 16:49:10.779446 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:10.779459 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:10.779533 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:10.779661 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:10.779662 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:10.779885 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:10.779951 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:10.779953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.780648 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.780649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:10.781205 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:10.781207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.781540 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.781541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:10.781620 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:10.782351 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:10.782359 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:10.782770 (+ 411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:10.784876 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:10.784877 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:10.784878 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:10.785054 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:10.785324 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:10.785325 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:10.785325 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:10.785396 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:10.785675 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:10.785676 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:10.785730 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 16:49:10.785792 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 16:49:10.785923 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":171,"threads_started":1}
I20260619 16:49:10.787499 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 16:49:10 dist-test-slave-jsw9 krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 16:49:10 dist-test-slave-jsw9 krb5kdc[2081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:10.816883 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:10.807535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40656 (local address 127.0.98.62:46337)
0619 16:49:10.807620 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:10.807623 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:10.807633 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:49:10.807664 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:10.807664 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:10.807675 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:10.807740 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:10.808582 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.809183 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.810120 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.810443 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.810514 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:10.813757 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:10.813759 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:10.813759 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:10.813769 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:10.815479 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:10.815909 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:10.815910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:10.815911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:10.815986 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:10.816291 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:10.816292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:10.816292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:10.816451 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:10.816549 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:10.816707 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 16:49:10.816799 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:49:10.819051 2125 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:49:10.821167 2125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:49:10.829697 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:10.824659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42588 (local address 127.0.98.2:41853)
0619 16:49:10.825392 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:10.825396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:10.825407 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:49:10.825554 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:10.825556 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:10.825574 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:10.825644 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:10.825755 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.826648 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.828376 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.829157 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.829252 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:10.829331 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:10.829485 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 16:49:10.829539 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":87,"threads_started":1}
I20260619 16:49:10.829620 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:10.824537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41853 (local address 127.0.0.1:42588)
0619 16:49:10.825347 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:10.825359 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:10.825458 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:10.825652 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:10.825653 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:10.825655 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:10.825705 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:10.825707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.826722 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.826723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:10.828171 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:10.828173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.829265 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.829266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:10.829340 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:10.829355 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 16:49:10.829437 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 16:49:10.829510 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"mutex_wait_us":51,"thread_start_us":128,"threads_started":1}
I20260619 16:49:10.830677 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:10.824531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39321 (local address 127.0.0.1:40674)
0619 16:49:10.824752 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:10.824813 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:10.824916 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:10.825374 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:10.825378 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:10.825381 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:10.825472 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:10.825474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.826954 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.826955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:10.828170 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:10.828173 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.830391 (+ 2218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.830392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:10.830476 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:10.830481 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:49:10.830538 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 16:49:10.830591 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":6,"thread_start_us":74,"threads_started":1}
I20260619 16:49:10.830758 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:10.824841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40674 (local address 127.0.98.1:39321)
0619 16:49:10.825076 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:10.825079 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:10.825088 (+ 9us) server_negotiation.cc:407] Connection header received
0619 16:49:10.825136 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:10.825138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:10.825155 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:10.825249 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:10.825570 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.826863 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.829380 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.830209 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.830288 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:10.830357 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:10.830561 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 16:49:10.830613 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":122,"threads_started":1}
I20260619 16:49:10.831038 2403 tablet_service.cc:1467] Processing CreateTablet for tablet 6f8a1797cb454ee4bf034de3e0debd65 (DEFAULT_TABLE table=test-table [id=b2046d758b2d4508bb3160a65297b23d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:10.831238 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:10.824532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32883 (local address 127.0.0.1:52762)
0619 16:49:10.824834 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:10.824846 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:10.824907 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:10.825472 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:10.825472 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:10.825474 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:10.825521 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:10.825522 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.828273 (+ 2751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.828274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:10.828994 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:10.828995 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.831020 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.831020 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:10.831074 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:10.831077 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:49:10.831116 (+ 39us) client_negotiation.cc:240] Negotiation successful
0619 16:49:10.831164 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":2,"spinlock_wait_cycles":2432,"thread_start_us":105,"threads_started":1}
I20260619 16:49:10.831509 2403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f8a1797cb454ee4bf034de3e0debd65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:10.831638 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:10.824826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52762 (local address 127.0.98.3:32883)
0619 16:49:10.825063 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:10.825069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:10.825083 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:49:10.825136 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:10.825138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:10.825155 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:10.825249 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:10.825603 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.826863 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.829098 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.830707 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.830782 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:10.831446 (+ 664us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:10.831480 (+ 34us) server_negotiation.cc:299] Negotiation successful
0619 16:49:10.831530 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":98,"threads_started":1}
I20260619 16:49:10.831578 2263 tablet_service.cc:1467] Processing CreateTablet for tablet 6f8a1797cb454ee4bf034de3e0debd65 (DEFAULT_TABLE table=test-table [id=b2046d758b2d4508bb3160a65297b23d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:10.831965 2263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f8a1797cb454ee4bf034de3e0debd65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:10.832304 2543 tablet_service.cc:1467] Processing CreateTablet for tablet 6f8a1797cb454ee4bf034de3e0debd65 (DEFAULT_TABLE table=test-table [id=b2046d758b2d4508bb3160a65297b23d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:10.832670 2543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f8a1797cb454ee4bf034de3e0debd65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:10.834969 2632 tablet_bootstrap.cc:492] T 6f8a1797cb454ee4bf034de3e0debd65 P bc51e846ab01481e823acbcb5b5a9211: Bootstrap starting.
I20260619 16:49:10.835736 2633 tablet_bootstrap.cc:492] T 6f8a1797cb454ee4bf034de3e0debd65 P 2bca37447f014aeca45e6117d296bc8e: Bootstrap starting.
I20260619 16:49:10.835915 2632 tablet_bootstrap.cc:654] T 6f8a1797cb454ee4bf034de3e0debd65 P bc51e846ab01481e823acbcb5b5a9211: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:10.836151 2634 tablet_bootstrap.cc:492] T 6f8a1797cb454ee4bf034de3e0debd65 P 4a17f838c5e94a6d87248e8ca5ea9ce0: Bootstrap starting.
I20260619 16:49:10.836308 2632 log.cc:824] T 6f8a1797cb454ee4bf034de3e0debd65 P bc51e846ab01481e823acbcb5b5a9211: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:10.836725 2633 tablet_bootstrap.cc:654] T 6f8a1797cb454ee4bf034de3e0debd65 P 2bca37447f014aeca45e6117d296bc8e: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:10.837052 2633 log.cc:824] T 6f8a1797cb454ee4bf034de3e0debd65 P 2bca37447f014aeca45e6117d296bc8e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:10.837090 2634 tablet_bootstrap.cc:654] T 6f8a1797cb454ee4bf034de3e0debd65 P 4a17f838c5e94a6d87248e8ca5ea9ce0: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:10.837393 2632 tablet_bootstrap.cc:492] T 6f8a1797cb454ee4bf034de3e0debd65 P bc51e846ab01481e823acbcb5b5a9211: No bootstrap required, opened a new log
I20260619 16:49:10.837430 2634 log.cc:824] T 6f8a1797cb454ee4bf034de3e0debd65 P 4a17f838c5e94a6d87248e8ca5ea9ce0: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:10.837467 2632 ts_tablet_manager.cc:1397] T 6f8a1797cb454ee4bf034de3e0debd65 P bc51e846ab01481e823acbcb5b5a9211: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:49:10.837893 2633 tablet_bootstrap.cc:492] T 6f8a1797cb454ee4bf034de3e0debd65 P 2bca37447f014aeca45e6117d296bc8e: No bootstrap required, opened a new log
I20260619 16:49:10.837980 2633 ts_tablet_manager.cc:1397] T 6f8a1797cb454ee4bf034de3e0debd65 P 2bca37447f014aeca45e6117d296bc8e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:49:10.838351 2634 tablet_bootstrap.cc:492] T 6f8a1797cb454ee4bf034de3e0debd65 P 4a17f838c5e94a6d87248e8ca5ea9ce0: No bootstrap required, opened a new log
I20260619 16:49:10.838426 2634 ts_tablet_manager.cc:1397] T 6f8a1797cb454ee4bf034de3e0debd65 P 4a17f838c5e94a6d87248e8ca5ea9ce0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:49:10.839648 2632 raft_consensus.cc:348] T 6f8a1797cb454ee4bf034de3e0debd65 P bc51e846ab01481e823acbcb5b5a9211 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } } peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } }
I20260619 16:49:10.839718 2633 raft_consensus.cc:348] T 6f8a1797cb454ee4bf034de3e0debd65 P 2bca37447f014aeca45e6117d296bc8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } } peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } }
I20260619 16:49:10.839857 2632 raft_consensus.cc:374] T 6f8a1797cb454ee4bf034de3e0debd65 P bc51e846ab01481e823acbcb5b5a9211 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:10.839864 2633 raft_consensus.cc:374] T 6f8a1797cb454ee4bf034de3e0debd65 P 2bca37447f014aeca45e6117d296bc8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:10.839895 2632 raft_consensus.cc:729] T 6f8a1797cb454ee4bf034de3e0debd65 P bc51e846ab01481e823acbcb5b5a9211 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc51e846ab01481e823acbcb5b5a9211, State: Initialized, Role: FOLLOWER
I20260619 16:49:10.839926 2633 raft_consensus.cc:729] T 6f8a1797cb454ee4bf034de3e0debd65 P 2bca37447f014aeca45e6117d296bc8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bca37447f014aeca45e6117d296bc8e, State: Initialized, Role: FOLLOWER
I20260619 16:49:10.840026 2632 consensus_queue.cc:260] T 6f8a1797cb454ee4bf034de3e0debd65 P bc51e846ab01481e823acbcb5b5a9211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } } peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } }
I20260619 16:49:10.840101 2633 consensus_queue.cc:260] T 6f8a1797cb454ee4bf034de3e0debd65 P 2bca37447f014aeca45e6117d296bc8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } } peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } }
I20260619 16:49:10.840360 2632 ts_tablet_manager.cc:1428] T 6f8a1797cb454ee4bf034de3e0debd65 P bc51e846ab01481e823acbcb5b5a9211: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 16:49:10.840389 2633 ts_tablet_manager.cc:1428] T 6f8a1797cb454ee4bf034de3e0debd65 P 2bca37447f014aeca45e6117d296bc8e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 16:49:10.840516 2329 heartbeater.cc:503] Master 127.0.98.62:46337 was elected leader, sending a full tablet report...
I20260619 16:49:10.840538 2469 heartbeater.cc:503] Master 127.0.98.62:46337 was elected leader, sending a full tablet report...
I20260619 16:49:10.840451 2634 raft_consensus.cc:348] T 6f8a1797cb454ee4bf034de3e0debd65 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } } peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } }
I20260619 16:49:10.840587 2634 raft_consensus.cc:374] T 6f8a1797cb454ee4bf034de3e0debd65 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:10.840617 2634 raft_consensus.cc:729] T 6f8a1797cb454ee4bf034de3e0debd65 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a17f838c5e94a6d87248e8ca5ea9ce0, State: Initialized, Role: FOLLOWER
I20260619 16:49:10.840741 2634 consensus_queue.cc:260] T 6f8a1797cb454ee4bf034de3e0debd65 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } } peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } }
I20260619 16:49:10.841030 2609 heartbeater.cc:503] Master 127.0.98.62:46337 was elected leader, sending a full tablet report...
I20260619 16:49:10.841264 2634 ts_tablet_manager.cc:1428] T 6f8a1797cb454ee4bf034de3e0debd65 P 4a17f838c5e94a6d87248e8ca5ea9ce0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260619 16:49:10.863467 2330 tablet.cc:2369] T 6f8a1797cb454ee4bf034de3e0debd65 P bc51e846ab01481e823acbcb5b5a9211: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:49:10.911428 2638 raft_consensus.cc:482] T 6f8a1797cb454ee4bf034de3e0debd65 P bc51e846ab01481e823acbcb5b5a9211 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:10.911598 2638 raft_consensus.cc:504] T 6f8a1797cb454ee4bf034de3e0debd65 P bc51e846ab01481e823acbcb5b5a9211 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } } peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } }
I20260619 16:49:10.912053 2638 leader_election.cc:288] T 6f8a1797cb454ee4bf034de3e0debd65 P bc51e846ab01481e823acbcb5b5a9211 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a17f838c5e94a6d87248e8ca5ea9ce0 (127.0.98.3:32883), 2bca37447f014aeca45e6117d296bc8e (127.0.98.2:41853)
I20260619 16:49:10.916440 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:10.912339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51901 (local address 127.0.98.3:32883)
0619 16:49:10.912431 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:10.912433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:10.912531 (+ 98us) server_negotiation.cc:407] Connection header received
0619 16:49:10.912595 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:10.912596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:10.912610 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:10.912673 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:10.912849 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.913664 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.915343 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.915934 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.916021 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:10.916039 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:10.916273 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 16:49:10.916325 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 16:49:10.916414 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:10.912363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60857 (local address 127.0.98.2:41853)
0619 16:49:10.912448 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:10.912450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:10.912531 (+ 81us) server_negotiation.cc:407] Connection header received
0619 16:49:10.912595 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:10.912596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:10.912610 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:10.912673 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:10.912856 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.913635 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.915320 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.915995 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.916061 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:10.916072 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:10.916275 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 16:49:10.916325 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:49:10.916458 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:10.912290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32883 (local address 127.0.98.1:51901)
0619 16:49:10.912433 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:10.912446 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:10.912531 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:10.912686 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:10.912687 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:10.912690 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:10.912781 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:10.912785 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.913746 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.913747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:10.915207 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:10.915210 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.916076 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.916077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:10.916179 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:10.916190 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 16:49:10.916257 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 16:49:10.916317 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":82,"thread_start_us":71,"threads_started":1}
I20260619 16:49:10.916466 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:10.912257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41853 (local address 127.0.98.1:60857)
0619 16:49:10.912434 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:10.912446 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:10.912531 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:10.912686 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:10.912687 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:10.912690 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:10.912796 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:10.912797 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.913761 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.913762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:10.915207 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:10.915210 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.916080 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.916081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:10.916179 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:10.916189 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 16:49:10.916252 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 16:49:10.916317 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":109,"thread_start_us":106,"threads_started":1}
I20260619 16:49:10.916858 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8a1797cb454ee4bf034de3e0debd65" candidate_uuid: "bc51e846ab01481e823acbcb5b5a9211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" is_pre_election: true
I20260619 16:49:10.917083 2563 raft_consensus.cc:2454] T 6f8a1797cb454ee4bf034de3e0debd65 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc51e846ab01481e823acbcb5b5a9211 in term 0.
I20260619 16:49:10.917079 2410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8a1797cb454ee4bf034de3e0debd65" candidate_uuid: "bc51e846ab01481e823acbcb5b5a9211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bca37447f014aeca45e6117d296bc8e" is_pre_election: true
I20260619 16:49:10.917241 2410 raft_consensus.cc:2454] T 6f8a1797cb454ee4bf034de3e0debd65 P 2bca37447f014aeca45e6117d296bc8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc51e846ab01481e823acbcb5b5a9211 in term 0.
I20260619 16:49:10.917359 2214 leader_election.cc:302] T 6f8a1797cb454ee4bf034de3e0debd65 P bc51e846ab01481e823acbcb5b5a9211 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a17f838c5e94a6d87248e8ca5ea9ce0, bc51e846ab01481e823acbcb5b5a9211; no voters:
I20260619 16:49:10.917505 2638 raft_consensus.cc:2781] T 6f8a1797cb454ee4bf034de3e0debd65 P bc51e846ab01481e823acbcb5b5a9211 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:49:10.917563 2638 raft_consensus.cc:482] T 6f8a1797cb454ee4bf034de3e0debd65 P bc51e846ab01481e823acbcb5b5a9211 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:49:10.917596 2638 raft_consensus.cc:3037] T 6f8a1797cb454ee4bf034de3e0debd65 P bc51e846ab01481e823acbcb5b5a9211 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:10.918663 2638 raft_consensus.cc:504] T 6f8a1797cb454ee4bf034de3e0debd65 P bc51e846ab01481e823acbcb5b5a9211 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } } peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } }
I20260619 16:49:10.918813 2638 leader_election.cc:288] T 6f8a1797cb454ee4bf034de3e0debd65 P bc51e846ab01481e823acbcb5b5a9211 [CANDIDATE]: Term 1 election: Requested vote from peers 4a17f838c5e94a6d87248e8ca5ea9ce0 (127.0.98.3:32883), 2bca37447f014aeca45e6117d296bc8e (127.0.98.2:41853)
I20260619 16:49:10.918957 2410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8a1797cb454ee4bf034de3e0debd65" candidate_uuid: "bc51e846ab01481e823acbcb5b5a9211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bca37447f014aeca45e6117d296bc8e"
I20260619 16:49:10.918957 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8a1797cb454ee4bf034de3e0debd65" candidate_uuid: "bc51e846ab01481e823acbcb5b5a9211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0"
I20260619 16:49:10.919039 2410 raft_consensus.cc:3037] T 6f8a1797cb454ee4bf034de3e0debd65 P 2bca37447f014aeca45e6117d296bc8e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:10.919046 2563 raft_consensus.cc:3037] T 6f8a1797cb454ee4bf034de3e0debd65 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:10.919822 2410 raft_consensus.cc:2454] T 6f8a1797cb454ee4bf034de3e0debd65 P 2bca37447f014aeca45e6117d296bc8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc51e846ab01481e823acbcb5b5a9211 in term 1.
I20260619 16:49:10.919871 2563 raft_consensus.cc:2454] T 6f8a1797cb454ee4bf034de3e0debd65 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc51e846ab01481e823acbcb5b5a9211 in term 1.
I20260619 16:49:10.920056 2215 leader_election.cc:302] T 6f8a1797cb454ee4bf034de3e0debd65 P bc51e846ab01481e823acbcb5b5a9211 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bca37447f014aeca45e6117d296bc8e, bc51e846ab01481e823acbcb5b5a9211; no voters:
I20260619 16:49:10.920159 2638 raft_consensus.cc:2781] T 6f8a1797cb454ee4bf034de3e0debd65 P bc51e846ab01481e823acbcb5b5a9211 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:10.920290 2638 raft_consensus.cc:686] T 6f8a1797cb454ee4bf034de3e0debd65 P bc51e846ab01481e823acbcb5b5a9211 [term 1 LEADER]: Becoming Leader. State: Replica: bc51e846ab01481e823acbcb5b5a9211, State: Running, Role: LEADER
I20260619 16:49:10.920390 2638 consensus_queue.cc:237] T 6f8a1797cb454ee4bf034de3e0debd65 P bc51e846ab01481e823acbcb5b5a9211 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } } peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } }
I20260619 16:49:10.921324 2125 catalog_manager.cc:5495] T 6f8a1797cb454ee4bf034de3e0debd65 P bc51e846ab01481e823acbcb5b5a9211 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc51e846ab01481e823acbcb5b5a9211 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "bc51e846ab01481e823acbcb5b5a9211" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } health_report { overall_health: UNKNOWN } } }
Jun 19 16:49:10 dist-test-slave-jsw9 krb5kdc[2081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:10.967303 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:10.959370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40662 (local address 127.0.98.62:46337)
0619 16:49:10.959444 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:10.959446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:10.959542 (+ 96us) server_negotiation.cc:407] Connection header received
0619 16:49:10.959601 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:10.959601 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:10.959610 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:10.959661 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:10.960217 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.960774 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.961510 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.961703 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.961771 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:10.964461 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:10.964463 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:10.964464 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:10.964471 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:10.966045 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:10.966523 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:10.966524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:10.966525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:10.966582 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:10.966744 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:10.966745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:10.966746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:10.966879 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:10.966943 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:10.967088 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 16:49:10.967184 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 16:49:10.967381 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:10.959181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46337 (local address 127.0.0.1:40662)
0619 16:49:10.959458 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:10.959474 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:10.959556 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:10.959673 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:10.959674 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:10.960058 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:10.960120 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:10.960124 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.960857 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.960858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:10.961419 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:10.961420 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:10.961809 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:10.961810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:10.961875 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:10.962607 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:10.962613 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:10.964317 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:10.966141 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:10.966155 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:10.966158 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:10.966455 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:10.966636 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:10.966637 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:10.966637 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:10.966696 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:10.966931 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:10.966941 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:10.967000 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 16:49:10.967060 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 16:49:10.967261 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":75,"thread_start_us":117,"threads_started":1}
I20260619 16:49:10.970253 2125 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40662:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 16:49:10.970542 2125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:49:10.972368 2403 tablet_service.cc:1467] Processing CreateTablet for tablet e4a79a38835040c0ba69f9999d70c633 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f3c7ba5e9aa04f8e821881c92dfe6e2d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:49:10.972373 2263 tablet_service.cc:1467] Processing CreateTablet for tablet e4a79a38835040c0ba69f9999d70c633 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f3c7ba5e9aa04f8e821881c92dfe6e2d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:49:10.972378 2543 tablet_service.cc:1467] Processing CreateTablet for tablet e4a79a38835040c0ba69f9999d70c633 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f3c7ba5e9aa04f8e821881c92dfe6e2d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:49:10.972561 2543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4a79a38835040c0ba69f9999d70c633. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:10.972560 2263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4a79a38835040c0ba69f9999d70c633. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:10.972561 2403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4a79a38835040c0ba69f9999d70c633. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:10.974506 2633 tablet_bootstrap.cc:492] T e4a79a38835040c0ba69f9999d70c633 P 2bca37447f014aeca45e6117d296bc8e: Bootstrap starting.
I20260619 16:49:10.974568 2632 tablet_bootstrap.cc:492] T e4a79a38835040c0ba69f9999d70c633 P bc51e846ab01481e823acbcb5b5a9211: Bootstrap starting.
I20260619 16:49:10.974578 2634 tablet_bootstrap.cc:492] T e4a79a38835040c0ba69f9999d70c633 P 4a17f838c5e94a6d87248e8ca5ea9ce0: Bootstrap starting.
I20260619 16:49:10.975134 2632 tablet_bootstrap.cc:654] T e4a79a38835040c0ba69f9999d70c633 P bc51e846ab01481e823acbcb5b5a9211: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:10.975160 2634 tablet_bootstrap.cc:654] T e4a79a38835040c0ba69f9999d70c633 P 4a17f838c5e94a6d87248e8ca5ea9ce0: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:10.975235 2633 tablet_bootstrap.cc:654] T e4a79a38835040c0ba69f9999d70c633 P 2bca37447f014aeca45e6117d296bc8e: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:10.976070 2632 tablet_bootstrap.cc:492] T e4a79a38835040c0ba69f9999d70c633 P bc51e846ab01481e823acbcb5b5a9211: No bootstrap required, opened a new log
I20260619 16:49:10.976141 2632 ts_tablet_manager.cc:1397] T e4a79a38835040c0ba69f9999d70c633 P bc51e846ab01481e823acbcb5b5a9211: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:49:10.976305 2633 tablet_bootstrap.cc:492] T e4a79a38835040c0ba69f9999d70c633 P 2bca37447f014aeca45e6117d296bc8e: No bootstrap required, opened a new log
I20260619 16:49:10.976372 2633 ts_tablet_manager.cc:1397] T e4a79a38835040c0ba69f9999d70c633 P 2bca37447f014aeca45e6117d296bc8e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:49:10.976336 2632 raft_consensus.cc:348] T e4a79a38835040c0ba69f9999d70c633 P bc51e846ab01481e823acbcb5b5a9211 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } }
I20260619 16:49:10.976431 2632 raft_consensus.cc:374] T e4a79a38835040c0ba69f9999d70c633 P bc51e846ab01481e823acbcb5b5a9211 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:10.976451 2632 raft_consensus.cc:729] T e4a79a38835040c0ba69f9999d70c633 P bc51e846ab01481e823acbcb5b5a9211 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc51e846ab01481e823acbcb5b5a9211, State: Initialized, Role: FOLLOWER
I20260619 16:49:10.976447 2634 tablet_bootstrap.cc:492] T e4a79a38835040c0ba69f9999d70c633 P 4a17f838c5e94a6d87248e8ca5ea9ce0: No bootstrap required, opened a new log
I20260619 16:49:10.976500 2634 ts_tablet_manager.cc:1397] T e4a79a38835040c0ba69f9999d70c633 P 4a17f838c5e94a6d87248e8ca5ea9ce0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:49:10.976500 2632 consensus_queue.cc:260] T e4a79a38835040c0ba69f9999d70c633 P bc51e846ab01481e823acbcb5b5a9211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } }
I20260619 16:49:10.976595 2632 ts_tablet_manager.cc:1428] T e4a79a38835040c0ba69f9999d70c633 P bc51e846ab01481e823acbcb5b5a9211: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 16:49:10.976572 2633 raft_consensus.cc:348] T e4a79a38835040c0ba69f9999d70c633 P 2bca37447f014aeca45e6117d296bc8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } }
I20260619 16:49:10.976650 2633 raft_consensus.cc:374] T e4a79a38835040c0ba69f9999d70c633 P 2bca37447f014aeca45e6117d296bc8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:10.976675 2633 raft_consensus.cc:729] T e4a79a38835040c0ba69f9999d70c633 P 2bca37447f014aeca45e6117d296bc8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bca37447f014aeca45e6117d296bc8e, State: Initialized, Role: FOLLOWER
I20260619 16:49:10.976667 2634 raft_consensus.cc:348] T e4a79a38835040c0ba69f9999d70c633 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } }
I20260619 16:49:10.976717 2633 consensus_queue.cc:260] T e4a79a38835040c0ba69f9999d70c633 P 2bca37447f014aeca45e6117d296bc8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } }
I20260619 16:49:10.976758 2634 raft_consensus.cc:374] T e4a79a38835040c0ba69f9999d70c633 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:10.976784 2634 raft_consensus.cc:729] T e4a79a38835040c0ba69f9999d70c633 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a17f838c5e94a6d87248e8ca5ea9ce0, State: Initialized, Role: FOLLOWER
I20260619 16:49:10.976763 2638 tablet_replica.cc:439] T e4a79a38835040c0ba69f9999d70c633 P bc51e846ab01481e823acbcb5b5a9211: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } } }
I20260619 16:49:10.976805 2633 ts_tablet_manager.cc:1428] T e4a79a38835040c0ba69f9999d70c633 P 2bca37447f014aeca45e6117d296bc8e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 16:49:10.976840 2638 tablet_replica.cc:442] T e4a79a38835040c0ba69f9999d70c633 P bc51e846ab01481e823acbcb5b5a9211: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:10.976825 2634 consensus_queue.cc:260] T e4a79a38835040c0ba69f9999d70c633 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } }
I20260619 16:49:10.976926 2634 ts_tablet_manager.cc:1428] T e4a79a38835040c0ba69f9999d70c633 P 4a17f838c5e94a6d87248e8ca5ea9ce0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:10.976984 2639 tablet_replica.cc:439] T e4a79a38835040c0ba69f9999d70c633 P 2bca37447f014aeca45e6117d296bc8e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } } }
I20260619 16:49:10.977087 2639 tablet_replica.cc:442] T e4a79a38835040c0ba69f9999d70c633 P 2bca37447f014aeca45e6117d296bc8e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:10.977254 2640 tablet_replica.cc:439] T e4a79a38835040c0ba69f9999d70c633 P 4a17f838c5e94a6d87248e8ca5ea9ce0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } } }
I20260619 16:49:10.977362 2640 tablet_replica.cc:442] T e4a79a38835040c0ba69f9999d70c633 P 4a17f838c5e94a6d87248e8ca5ea9ce0: This TxnStatusTablet replica's current role is: FOLLOWER
W20260619 16:49:11.025400 2610 tablet.cc:2369] T 6f8a1797cb454ee4bf034de3e0debd65 P 4a17f838c5e94a6d87248e8ca5ea9ce0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:49:11.075618 2470 tablet.cc:2369] T 6f8a1797cb454ee4bf034de3e0debd65 P 2bca37447f014aeca45e6117d296bc8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:49:11.104507 2639 raft_consensus.cc:482] T e4a79a38835040c0ba69f9999d70c633 P 2bca37447f014aeca45e6117d296bc8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:11.104648 2639 raft_consensus.cc:504] T e4a79a38835040c0ba69f9999d70c633 P 2bca37447f014aeca45e6117d296bc8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } }
I20260619 16:49:11.105096 2639 leader_election.cc:288] T e4a79a38835040c0ba69f9999d70c633 P 2bca37447f014aeca45e6117d296bc8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc51e846ab01481e823acbcb5b5a9211 (127.0.98.1:39321), 4a17f838c5e94a6d87248e8ca5ea9ce0 (127.0.98.3:32883)
I20260619 16:49:11.109933 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.105451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35813 (local address 127.0.98.1:39321)
0619 16:49:11.105512 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:11.105514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:11.105631 (+ 117us) server_negotiation.cc:407] Connection header received
0619 16:49:11.105698 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:11.105699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:11.105711 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:11.105768 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:11.105997 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.107012 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.108863 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.109514 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.109598 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.109612 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:11.109789 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 16:49:11.109838 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 16:49:11.109951 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.105405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50709 (local address 127.0.98.3:32883)
0619 16:49:11.105500 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:11.105503 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:11.105630 (+ 127us) server_negotiation.cc:407] Connection header received
0619 16:49:11.105696 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:11.105697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:11.105709 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:11.105768 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:11.105997 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.106986 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.108861 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.109519 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.109598 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.109612 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:11.109789 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 16:49:11.109840 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:49:11.109983 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.105311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32883 (local address 127.0.98.2:50709)
0619 16:49:11.105545 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:11.105554 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:11.105633 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:11.105790 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:11.105791 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:11.105794 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:11.105926 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.105929 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.107132 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.107134 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.108739 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.108745 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.109609 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.109610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.109704 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.109711 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 16:49:11.109775 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 16:49:11.109847 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":168,"thread_start_us":103,"threads_started":1}
I20260619 16:49:11.109985 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.105311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39321 (local address 127.0.98.2:35813)
0619 16:49:11.105500 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:11.105513 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:11.105633 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:11.105790 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:11.105791 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:11.105795 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:11.105926 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.105929 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.107132 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.107134 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.108761 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.108763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.109608 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.109610 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.109715 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.109718 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:49:11.109775 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 16:49:11.109847 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":70,"thread_start_us":114,"threads_started":1}
I20260619 16:49:11.110297 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a79a38835040c0ba69f9999d70c633" candidate_uuid: "2bca37447f014aeca45e6117d296bc8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" is_pre_election: true
I20260619 16:49:11.110411 2563 raft_consensus.cc:2454] T e4a79a38835040c0ba69f9999d70c633 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bca37447f014aeca45e6117d296bc8e in term 0.
I20260619 16:49:11.110605 2354 leader_election.cc:302] T e4a79a38835040c0ba69f9999d70c633 P 2bca37447f014aeca45e6117d296bc8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bca37447f014aeca45e6117d296bc8e, 4a17f838c5e94a6d87248e8ca5ea9ce0; no voters:
I20260619 16:49:11.110630 2283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a79a38835040c0ba69f9999d70c633" candidate_uuid: "2bca37447f014aeca45e6117d296bc8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc51e846ab01481e823acbcb5b5a9211" is_pre_election: true
I20260619 16:49:11.110736 2283 raft_consensus.cc:2454] T e4a79a38835040c0ba69f9999d70c633 P bc51e846ab01481e823acbcb5b5a9211 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bca37447f014aeca45e6117d296bc8e in term 0.
I20260619 16:49:11.110767 2639 raft_consensus.cc:2781] T e4a79a38835040c0ba69f9999d70c633 P 2bca37447f014aeca45e6117d296bc8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:49:11.110816 2639 raft_consensus.cc:482] T e4a79a38835040c0ba69f9999d70c633 P 2bca37447f014aeca45e6117d296bc8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:49:11.110845 2639 raft_consensus.cc:3037] T e4a79a38835040c0ba69f9999d70c633 P 2bca37447f014aeca45e6117d296bc8e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:11.111703 2639 raft_consensus.cc:504] T e4a79a38835040c0ba69f9999d70c633 P 2bca37447f014aeca45e6117d296bc8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } }
I20260619 16:49:11.111851 2639 leader_election.cc:288] T e4a79a38835040c0ba69f9999d70c633 P 2bca37447f014aeca45e6117d296bc8e [CANDIDATE]: Term 1 election: Requested vote from peers bc51e846ab01481e823acbcb5b5a9211 (127.0.98.1:39321), 4a17f838c5e94a6d87248e8ca5ea9ce0 (127.0.98.3:32883)
I20260619 16:49:11.112074 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a79a38835040c0ba69f9999d70c633" candidate_uuid: "2bca37447f014aeca45e6117d296bc8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0"
I20260619 16:49:11.112165 2563 raft_consensus.cc:3037] T e4a79a38835040c0ba69f9999d70c633 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:11.112182 2283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a79a38835040c0ba69f9999d70c633" candidate_uuid: "2bca37447f014aeca45e6117d296bc8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc51e846ab01481e823acbcb5b5a9211"
I20260619 16:49:11.112285 2283 raft_consensus.cc:3037] T e4a79a38835040c0ba69f9999d70c633 P bc51e846ab01481e823acbcb5b5a9211 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:11.112886 2563 raft_consensus.cc:2454] T e4a79a38835040c0ba69f9999d70c633 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bca37447f014aeca45e6117d296bc8e in term 1.
I20260619 16:49:11.113058 2354 leader_election.cc:302] T e4a79a38835040c0ba69f9999d70c633 P 2bca37447f014aeca45e6117d296bc8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bca37447f014aeca45e6117d296bc8e, 4a17f838c5e94a6d87248e8ca5ea9ce0; no voters:
I20260619 16:49:11.113062 2283 raft_consensus.cc:2454] T e4a79a38835040c0ba69f9999d70c633 P bc51e846ab01481e823acbcb5b5a9211 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bca37447f014aeca45e6117d296bc8e in term 1.
I20260619 16:49:11.113160 2639 raft_consensus.cc:2781] T e4a79a38835040c0ba69f9999d70c633 P 2bca37447f014aeca45e6117d296bc8e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:11.113353 2639 raft_consensus.cc:686] T e4a79a38835040c0ba69f9999d70c633 P 2bca37447f014aeca45e6117d296bc8e [term 1 LEADER]: Becoming Leader. State: Replica: 2bca37447f014aeca45e6117d296bc8e, State: Running, Role: LEADER
I20260619 16:49:11.113489 2639 consensus_queue.cc:237] T e4a79a38835040c0ba69f9999d70c633 P 2bca37447f014aeca45e6117d296bc8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } }
I20260619 16:49:11.113945 2656 tablet_replica.cc:439] T e4a79a38835040c0ba69f9999d70c633 P 2bca37447f014aeca45e6117d296bc8e: TxnStatusTablet state changed. Reason: New leader 2bca37447f014aeca45e6117d296bc8e. Latest consensus state: current_term: 1 leader_uuid: "2bca37447f014aeca45e6117d296bc8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } } }
I20260619 16:49:11.114043 2656 tablet_replica.cc:442] T e4a79a38835040c0ba69f9999d70c633 P 2bca37447f014aeca45e6117d296bc8e: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:49:11.114274 2658 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 16:49:11.114470 2658 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 16:49:11.114511 2125 catalog_manager.cc:5495] T e4a79a38835040c0ba69f9999d70c633 P 2bca37447f014aeca45e6117d296bc8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2bca37447f014aeca45e6117d296bc8e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2bca37447f014aeca45e6117d296bc8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } health_report { overall_health: UNKNOWN } } }
I20260619 16:49:11.129127 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.126393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42596 (local address 127.0.98.2:41853)
0619 16:49:11.126444 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:11.126445 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:11.126475 (+ 30us) server_negotiation.cc:407] Connection header received
0619 16:49:11.126577 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:11.126578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:11.126601 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:11.126657 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:11.126797 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.127435 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.128268 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.128475 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.128533 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.128720 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:11.128838 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:11.128918 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:11.128974 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 16:49:11.129001 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 16:49:11.129127 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.126332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41853 (local address 127.0.0.1:42596)
0619 16:49:11.126429 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:11.126442 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:11.126499 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:11.126665 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:11.126666 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:11.126667 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:49:11.126737 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.126739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.127549 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.127550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.128184 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.128188 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.128538 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.128539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.128616 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.128634 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:49:11.128915 (+ 281us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:49:11.128924 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 16:49:11.128969 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 16:49:11.129037 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 16:49:11.131245 2283 raft_consensus.cc:1261] T e4a79a38835040c0ba69f9999d70c633 P bc51e846ab01481e823acbcb5b5a9211 [term 1 FOLLOWER]: Refusing update from remote peer 2bca37447f014aeca45e6117d296bc8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:11.131336 2563 raft_consensus.cc:1261] T e4a79a38835040c0ba69f9999d70c633 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 1 FOLLOWER]: Refusing update from remote peer 2bca37447f014aeca45e6117d296bc8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:11.131597 2639 consensus_queue.cc:1035] T e4a79a38835040c0ba69f9999d70c633 P 2bca37447f014aeca45e6117d296bc8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:11.131767 2656 consensus_queue.cc:1035] T e4a79a38835040c0ba69f9999d70c633 P 2bca37447f014aeca45e6117d296bc8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:11.133184 2638 tablet_replica.cc:439] T e4a79a38835040c0ba69f9999d70c633 P bc51e846ab01481e823acbcb5b5a9211: TxnStatusTablet state changed. Reason: New leader 2bca37447f014aeca45e6117d296bc8e. Latest consensus state: current_term: 1 leader_uuid: "2bca37447f014aeca45e6117d296bc8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } } }
I20260619 16:49:11.133291 2638 tablet_replica.cc:442] T e4a79a38835040c0ba69f9999d70c633 P bc51e846ab01481e823acbcb5b5a9211: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:11.133575 2640 tablet_replica.cc:439] T e4a79a38835040c0ba69f9999d70c633 P 4a17f838c5e94a6d87248e8ca5ea9ce0: TxnStatusTablet state changed. Reason: New leader 2bca37447f014aeca45e6117d296bc8e. Latest consensus state: current_term: 1 leader_uuid: "2bca37447f014aeca45e6117d296bc8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } } }
I20260619 16:49:11.133684 2640 tablet_replica.cc:442] T e4a79a38835040c0ba69f9999d70c633 P 4a17f838c5e94a6d87248e8ca5ea9ce0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:11.134516 2656 tablet_replica.cc:439] T e4a79a38835040c0ba69f9999d70c633 P 2bca37447f014aeca45e6117d296bc8e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2bca37447f014aeca45e6117d296bc8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } } }
I20260619 16:49:11.134719 2656 tablet_replica.cc:442] T e4a79a38835040c0ba69f9999d70c633 P 2bca37447f014aeca45e6117d296bc8e: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:49:11.134742 2638 tablet_replica.cc:439] T e4a79a38835040c0ba69f9999d70c633 P bc51e846ab01481e823acbcb5b5a9211: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2bca37447f014aeca45e6117d296bc8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } } }
I20260619 16:49:11.134843 2638 tablet_replica.cc:442] T e4a79a38835040c0ba69f9999d70c633 P bc51e846ab01481e823acbcb5b5a9211: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:11.135478 2662 mvcc.cc:205] Tried to move back new op lower bound from 7298612228630691840 to 7298612228561424384. Current Snapshot: MvccSnapshot[applied={T|T < 7298612228630691840 or (T in {7298612228630691840})}]
I20260619 16:49:11.136214 2639 tablet_replica.cc:439] T e4a79a38835040c0ba69f9999d70c633 P 2bca37447f014aeca45e6117d296bc8e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2bca37447f014aeca45e6117d296bc8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } } }
I20260619 16:49:11.136327 2639 tablet_replica.cc:442] T e4a79a38835040c0ba69f9999d70c633 P 2bca37447f014aeca45e6117d296bc8e: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:49:11.136772 2640 tablet_replica.cc:439] T e4a79a38835040c0ba69f9999d70c633 P 4a17f838c5e94a6d87248e8ca5ea9ce0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2bca37447f014aeca45e6117d296bc8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } } }
I20260619 16:49:11.136889 2640 tablet_replica.cc:442] T e4a79a38835040c0ba69f9999d70c633 P 4a17f838c5e94a6d87248e8ca5ea9ce0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:11.145529 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.142704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40690 (local address 127.0.98.1:39321)
0619 16:49:11.142766 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:11.142768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:11.142778 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:49:11.142837 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:11.142838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:11.142858 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:11.142928 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:11.143089 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.143742 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.144587 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.144902 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.144964 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.145121 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:11.145293 (+ 172us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:11.145374 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:11.145398 (+ 24us) server_negotiation.cc:299] Negotiation successful
0619 16:49:11.145427 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 16:49:11.151391 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.148822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57637 (local address 127.0.98.2:41853)
0619 16:49:11.148902 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:11.148905 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:11.148952 (+ 47us) server_negotiation.cc:407] Connection header received
0619 16:49:11.149022 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:11.149022 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:11.149035 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:11.149105 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:11.149238 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.149789 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.150679 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.150901 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.150960 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.151135 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:11.151174 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:11.151215 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:11.151268 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 16:49:11.151292 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 16:49:11.151429 2673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.148723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41853 (local address 127.0.98.1:57637)
0619 16:49:11.148886 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:11.148899 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:11.148970 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:11.149107 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:11.149108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:11.149110 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:49:11.149190 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.149192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.149853 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.149854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.150598 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.150601 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.150965 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.150967 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.151031 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.151046 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:49:11.151224 (+ 178us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:49:11.151226 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 16:49:11.151266 (+ 40us) client_negotiation.cc:240] Negotiation successful
0619 16:49:11.151317 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
I20260619 16:49:11.154493 2563 raft_consensus.cc:1261] T 6f8a1797cb454ee4bf034de3e0debd65 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 1 FOLLOWER]: Refusing update from remote peer bc51e846ab01481e823acbcb5b5a9211: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:11.154522 2410 raft_consensus.cc:1261] T 6f8a1797cb454ee4bf034de3e0debd65 P 2bca37447f014aeca45e6117d296bc8e [term 1 FOLLOWER]: Refusing update from remote peer bc51e846ab01481e823acbcb5b5a9211: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:11.154817 2638 consensus_queue.cc:1035] T 6f8a1797cb454ee4bf034de3e0debd65 P bc51e846ab01481e823acbcb5b5a9211 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:11.155049 2643 consensus_queue.cc:1035] T 6f8a1797cb454ee4bf034de3e0debd65 P bc51e846ab01481e823acbcb5b5a9211 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:11.157671 2661 mvcc.cc:205] Tried to move back new op lower bound from 7298612228726444032 to 7298612227770716160. Current Snapshot: MvccSnapshot[applied={T|T < 7298612228726444032}]
I20260619 16:49:11.164361 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.161665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49393 (local address 127.0.98.1:39321)
0619 16:49:11.161711 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:11.161714 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:11.161868 (+ 154us) server_negotiation.cc:407] Connection header received
0619 16:49:11.161962 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:11.161962 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:11.161977 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:11.162023 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:11.162171 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.162718 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.163568 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.163839 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.163912 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.164100 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:11.164141 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:11.164179 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:11.164244 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 16:49:11.164271 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 16:49:11.164409 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.161615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39321 (local address 127.0.98.2:49393)
0619 16:49:11.161802 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:11.161814 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:11.161895 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:11.162024 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:11.162024 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:11.162026 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:49:11.162119 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.162121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.162795 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.162796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.163499 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.163503 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.163906 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.163908 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.163990 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.164012 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:49:11.164190 (+ 178us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:49:11.164192 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 16:49:11.164240 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 16:49:11.164289 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":86,"threads_started":1}
I20260619 16:49:11.175868 2125 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:11.176020 2125 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:11.177028 2125 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8990279310d642648682b1df509b9968: Sending DeleteTablet for 3 replicas of tablet 6f8a1797cb454ee4bf034de3e0debd65
I20260619 16:49:11.177574 2543 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f8a1797cb454ee4bf034de3e0debd65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52762
I20260619 16:49:11.177717 2263 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f8a1797cb454ee4bf034de3e0debd65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40674
I20260619 16:49:11.178032 2682 tablet_replica.cc:330] T 6f8a1797cb454ee4bf034de3e0debd65 P bc51e846ab01481e823acbcb5b5a9211: stopping tablet replica
I20260619 16:49:11.178220 2682 raft_consensus.cc:2229] T 6f8a1797cb454ee4bf034de3e0debd65 P bc51e846ab01481e823acbcb5b5a9211 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:49:11.178170 2403 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f8a1797cb454ee4bf034de3e0debd65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42588
I20260619 16:49:11.178457 2682 raft_consensus.cc:2258] T 6f8a1797cb454ee4bf034de3e0debd65 P bc51e846ab01481e823acbcb5b5a9211 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:11.178470 2125 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:49:11.178659 2125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:49:11.179839 2683 tablet_replica.cc:330] T 6f8a1797cb454ee4bf034de3e0debd65 P 2bca37447f014aeca45e6117d296bc8e: stopping tablet replica
I20260619 16:49:11.179983 2683 raft_consensus.cc:2229] T 6f8a1797cb454ee4bf034de3e0debd65 P 2bca37447f014aeca45e6117d296bc8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:11.180039 2683 raft_consensus.cc:2258] T 6f8a1797cb454ee4bf034de3e0debd65 P 2bca37447f014aeca45e6117d296bc8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:11.180249 2403 tablet_service.cc:1467] Processing CreateTablet for tablet 3249e60a58ef4fa686ce736e8ebc7c19 (DEFAULT_TABLE table=test-table [id=db40df6aa6e94126a69700458491a5c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:11.180404 2403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3249e60a58ef4fa686ce736e8ebc7c19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:11.180552 2683 ts_tablet_manager.cc:1905] T 6f8a1797cb454ee4bf034de3e0debd65 P 2bca37447f014aeca45e6117d296bc8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:11.182042 2543 tablet_service.cc:1467] Processing CreateTablet for tablet 3249e60a58ef4fa686ce736e8ebc7c19 (DEFAULT_TABLE table=test-table [id=db40df6aa6e94126a69700458491a5c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:11.182222 2543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3249e60a58ef4fa686ce736e8ebc7c19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:11.182518 2683 ts_tablet_manager.cc:1918] T 6f8a1797cb454ee4bf034de3e0debd65 P 2bca37447f014aeca45e6117d296bc8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:49:11.182587 2683 log.cc:1196] T 6f8a1797cb454ee4bf034de3e0debd65 P 2bca37447f014aeca45e6117d296bc8e: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-1/wal/wals/6f8a1797cb454ee4bf034de3e0debd65
I20260619 16:49:11.182623 2682 ts_tablet_manager.cc:1905] T 6f8a1797cb454ee4bf034de3e0debd65 P bc51e846ab01481e823acbcb5b5a9211: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:11.182868 2683 ts_tablet_manager.cc:1939] T 6f8a1797cb454ee4bf034de3e0debd65 P 2bca37447f014aeca45e6117d296bc8e: Deleting consensus metadata
I20260619 16:49:11.183724 2634 tablet_bootstrap.cc:492] T 3249e60a58ef4fa686ce736e8ebc7c19 P 4a17f838c5e94a6d87248e8ca5ea9ce0: Bootstrap starting.
I20260619 16:49:11.183746 2633 tablet_bootstrap.cc:492] T 3249e60a58ef4fa686ce736e8ebc7c19 P 2bca37447f014aeca45e6117d296bc8e: Bootstrap starting.
I20260619 16:49:11.184127 2633 tablet_bootstrap.cc:654] T 3249e60a58ef4fa686ce736e8ebc7c19 P 2bca37447f014aeca45e6117d296bc8e: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:11.184309 2634 tablet_bootstrap.cc:654] T 3249e60a58ef4fa686ce736e8ebc7c19 P 4a17f838c5e94a6d87248e8ca5ea9ce0: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:11.184648 2682 ts_tablet_manager.cc:1918] T 6f8a1797cb454ee4bf034de3e0debd65 P bc51e846ab01481e823acbcb5b5a9211: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:49:11.184725 2682 log.cc:1196] T 6f8a1797cb454ee4bf034de3e0debd65 P bc51e846ab01481e823acbcb5b5a9211: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-0/wal/wals/6f8a1797cb454ee4bf034de3e0debd65
I20260619 16:49:11.185057 2111 catalog_manager.cc:4841] TS 2bca37447f014aeca45e6117d296bc8e (127.0.98.2:41853): tablet 6f8a1797cb454ee4bf034de3e0debd65 (table test-table [id=b2046d758b2d4508bb3160a65297b23d]) successfully deleted
I20260619 16:49:11.185164 2682 ts_tablet_manager.cc:1939] T 6f8a1797cb454ee4bf034de3e0debd65 P bc51e846ab01481e823acbcb5b5a9211: Deleting consensus metadata
I20260619 16:49:11.185207 2634 tablet_bootstrap.cc:492] T 3249e60a58ef4fa686ce736e8ebc7c19 P 4a17f838c5e94a6d87248e8ca5ea9ce0: No bootstrap required, opened a new log
I20260619 16:49:11.185256 2634 ts_tablet_manager.cc:1397] T 3249e60a58ef4fa686ce736e8ebc7c19 P 4a17f838c5e94a6d87248e8ca5ea9ce0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:49:11.185446 2634 raft_consensus.cc:348] T 3249e60a58ef4fa686ce736e8ebc7c19 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } }
I20260619 16:49:11.185523 2634 raft_consensus.cc:374] T 3249e60a58ef4fa686ce736e8ebc7c19 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:11.185541 2634 raft_consensus.cc:729] T 3249e60a58ef4fa686ce736e8ebc7c19 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a17f838c5e94a6d87248e8ca5ea9ce0, State: Initialized, Role: FOLLOWER
I20260619 16:49:11.185588 2634 consensus_queue.cc:260] T 3249e60a58ef4fa686ce736e8ebc7c19 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } }
I20260619 16:49:11.185675 2634 ts_tablet_manager.cc:1428] T 3249e60a58ef4fa686ce736e8ebc7c19 P 4a17f838c5e94a6d87248e8ca5ea9ce0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 16:49:11.186069 2263 tablet_service.cc:1467] Processing CreateTablet for tablet 3249e60a58ef4fa686ce736e8ebc7c19 (DEFAULT_TABLE table=test-table [id=db40df6aa6e94126a69700458491a5c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:11.186224 2263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3249e60a58ef4fa686ce736e8ebc7c19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:11.186501 2633 tablet_bootstrap.cc:492] T 3249e60a58ef4fa686ce736e8ebc7c19 P 2bca37447f014aeca45e6117d296bc8e: No bootstrap required, opened a new log
I20260619 16:49:11.186556 2633 ts_tablet_manager.cc:1397] T 3249e60a58ef4fa686ce736e8ebc7c19 P 2bca37447f014aeca45e6117d296bc8e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 16:49:11.186718 2633 raft_consensus.cc:348] T 3249e60a58ef4fa686ce736e8ebc7c19 P 2bca37447f014aeca45e6117d296bc8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } }
I20260619 16:49:11.186795 2633 raft_consensus.cc:374] T 3249e60a58ef4fa686ce736e8ebc7c19 P 2bca37447f014aeca45e6117d296bc8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:11.186815 2633 raft_consensus.cc:729] T 3249e60a58ef4fa686ce736e8ebc7c19 P 2bca37447f014aeca45e6117d296bc8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bca37447f014aeca45e6117d296bc8e, State: Initialized, Role: FOLLOWER
I20260619 16:49:11.186859 2633 consensus_queue.cc:260] T 3249e60a58ef4fa686ce736e8ebc7c19 P 2bca37447f014aeca45e6117d296bc8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } }
I20260619 16:49:11.186954 2633 ts_tablet_manager.cc:1428] T 3249e60a58ef4fa686ce736e8ebc7c19 P 2bca37447f014aeca45e6117d296bc8e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:11.187680 2632 tablet_bootstrap.cc:492] T 3249e60a58ef4fa686ce736e8ebc7c19 P bc51e846ab01481e823acbcb5b5a9211: Bootstrap starting.
I20260619 16:49:11.188134 2632 tablet_bootstrap.cc:654] T 3249e60a58ef4fa686ce736e8ebc7c19 P bc51e846ab01481e823acbcb5b5a9211: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:11.188918 2113 catalog_manager.cc:4841] TS bc51e846ab01481e823acbcb5b5a9211 (127.0.98.1:39321): tablet 6f8a1797cb454ee4bf034de3e0debd65 (table test-table [id=b2046d758b2d4508bb3160a65297b23d]) successfully deleted
I20260619 16:49:11.189203 2632 tablet_bootstrap.cc:492] T 3249e60a58ef4fa686ce736e8ebc7c19 P bc51e846ab01481e823acbcb5b5a9211: No bootstrap required, opened a new log
I20260619 16:49:11.189252 2632 ts_tablet_manager.cc:1397] T 3249e60a58ef4fa686ce736e8ebc7c19 P bc51e846ab01481e823acbcb5b5a9211: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:49:11.189391 2632 raft_consensus.cc:348] T 3249e60a58ef4fa686ce736e8ebc7c19 P bc51e846ab01481e823acbcb5b5a9211 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } }
I20260619 16:49:11.189473 2632 raft_consensus.cc:374] T 3249e60a58ef4fa686ce736e8ebc7c19 P bc51e846ab01481e823acbcb5b5a9211 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:11.189492 2632 raft_consensus.cc:729] T 3249e60a58ef4fa686ce736e8ebc7c19 P bc51e846ab01481e823acbcb5b5a9211 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc51e846ab01481e823acbcb5b5a9211, State: Initialized, Role: FOLLOWER
I20260619 16:49:11.189535 2632 consensus_queue.cc:260] T 3249e60a58ef4fa686ce736e8ebc7c19 P bc51e846ab01481e823acbcb5b5a9211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } }
I20260619 16:49:11.189618 2632 ts_tablet_manager.cc:1428] T 3249e60a58ef4fa686ce736e8ebc7c19 P bc51e846ab01481e823acbcb5b5a9211: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:11.190431 2681 tablet_replica.cc:330] T 6f8a1797cb454ee4bf034de3e0debd65 P 4a17f838c5e94a6d87248e8ca5ea9ce0: stopping tablet replica
I20260619 16:49:11.190656 2681 raft_consensus.cc:2229] T 6f8a1797cb454ee4bf034de3e0debd65 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:11.190788 2681 raft_consensus.cc:2258] T 6f8a1797cb454ee4bf034de3e0debd65 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:11.191382 2681 ts_tablet_manager.cc:1905] T 6f8a1797cb454ee4bf034de3e0debd65 P 4a17f838c5e94a6d87248e8ca5ea9ce0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:11.193286 2681 ts_tablet_manager.cc:1918] T 6f8a1797cb454ee4bf034de3e0debd65 P 4a17f838c5e94a6d87248e8ca5ea9ce0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:49:11.193435 2681 log.cc:1196] T 6f8a1797cb454ee4bf034de3e0debd65 P 4a17f838c5e94a6d87248e8ca5ea9ce0: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-2/wal/wals/6f8a1797cb454ee4bf034de3e0debd65
I20260619 16:49:11.193735 2681 ts_tablet_manager.cc:1939] T 6f8a1797cb454ee4bf034de3e0debd65 P 4a17f838c5e94a6d87248e8ca5ea9ce0: Deleting consensus metadata
I20260619 16:49:11.194283 2110 catalog_manager.cc:4841] TS 4a17f838c5e94a6d87248e8ca5ea9ce0 (127.0.98.3:32883): tablet 6f8a1797cb454ee4bf034de3e0debd65 (table test-table [id=b2046d758b2d4508bb3160a65297b23d]) successfully deleted
I20260619 16:49:11.304754 2640 raft_consensus.cc:482] T 3249e60a58ef4fa686ce736e8ebc7c19 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:11.304922 2640 raft_consensus.cc:504] T 3249e60a58ef4fa686ce736e8ebc7c19 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } }
I20260619 16:49:11.305303 2640 leader_election.cc:288] T 3249e60a58ef4fa686ce736e8ebc7c19 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc51e846ab01481e823acbcb5b5a9211 (127.0.98.1:39321), 2bca37447f014aeca45e6117d296bc8e (127.0.98.2:41853)
I20260619 16:49:11.309368 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.305658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57363 (local address 127.0.98.1:39321)
0619 16:49:11.305711 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:11.305714 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:11.305780 (+ 66us) server_negotiation.cc:407] Connection header received
0619 16:49:11.305947 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:11.305947 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:11.305958 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:11.306032 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:11.306303 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.307289 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.308474 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.308955 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.309022 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.309036 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:11.309217 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 16:49:11.309271 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 16:49:11.309468 2688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.305504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41853 (local address 127.0.98.3:34961)
0619 16:49:11.305892 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:11.305904 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:11.306024 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:11.306241 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:11.306242 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:11.306244 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:11.306308 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.306310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.307041 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.307043 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.308346 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.308350 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.309198 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.309199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.309268 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.309272 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:49:11.309323 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 16:49:11.309382 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":3,"thread_start_us":242,"threads_started":1}
I20260619 16:49:11.309461 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.305632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34961 (local address 127.0.98.2:41853)
0619 16:49:11.305697 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:11.305700 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:11.306021 (+ 321us) server_negotiation.cc:407] Connection header received
0619 16:49:11.306103 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:11.306104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:11.306115 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:11.306153 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:11.306381 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.306920 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.308459 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.309099 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.309190 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.309204 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:11.309339 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 16:49:11.309388 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 16:49:11.309599 2687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.305504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39321 (local address 127.0.98.3:57363)
0619 16:49:11.305696 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:11.305708 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:11.305847 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:11.306057 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:11.306058 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:11.306061 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:11.306174 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.306177 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.307368 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.307369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.308398 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.308399 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.309032 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.309033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.309117 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.309122 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:49:11.309486 (+ 364us) client_negotiation.cc:240] Negotiation successful
0619 16:49:11.309531 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":106,"threads_started":1}
I20260619 16:49:11.309687 2410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3249e60a58ef4fa686ce736e8ebc7c19" candidate_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bca37447f014aeca45e6117d296bc8e" is_pre_election: true
I20260619 16:49:11.309808 2410 raft_consensus.cc:2454] T 3249e60a58ef4fa686ce736e8ebc7c19 P 2bca37447f014aeca45e6117d296bc8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a17f838c5e94a6d87248e8ca5ea9ce0 in term 0.
I20260619 16:49:11.309810 2283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3249e60a58ef4fa686ce736e8ebc7c19" candidate_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc51e846ab01481e823acbcb5b5a9211" is_pre_election: true
I20260619 16:49:11.310009 2283 raft_consensus.cc:2454] T 3249e60a58ef4fa686ce736e8ebc7c19 P bc51e846ab01481e823acbcb5b5a9211 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a17f838c5e94a6d87248e8ca5ea9ce0 in term 0.
I20260619 16:49:11.310161 2495 leader_election.cc:302] T 3249e60a58ef4fa686ce736e8ebc7c19 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bca37447f014aeca45e6117d296bc8e, 4a17f838c5e94a6d87248e8ca5ea9ce0; no voters:
I20260619 16:49:11.310354 2640 raft_consensus.cc:2781] T 3249e60a58ef4fa686ce736e8ebc7c19 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:49:11.310397 2640 raft_consensus.cc:482] T 3249e60a58ef4fa686ce736e8ebc7c19 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:49:11.310423 2640 raft_consensus.cc:3037] T 3249e60a58ef4fa686ce736e8ebc7c19 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:11.311240 2640 raft_consensus.cc:504] T 3249e60a58ef4fa686ce736e8ebc7c19 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } }
I20260619 16:49:11.311389 2640 leader_election.cc:288] T 3249e60a58ef4fa686ce736e8ebc7c19 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [CANDIDATE]: Term 1 election: Requested vote from peers bc51e846ab01481e823acbcb5b5a9211 (127.0.98.1:39321), 2bca37447f014aeca45e6117d296bc8e (127.0.98.2:41853)
I20260619 16:49:11.311565 2410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3249e60a58ef4fa686ce736e8ebc7c19" candidate_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bca37447f014aeca45e6117d296bc8e"
I20260619 16:49:11.311582 2283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3249e60a58ef4fa686ce736e8ebc7c19" candidate_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc51e846ab01481e823acbcb5b5a9211"
I20260619 16:49:11.311657 2410 raft_consensus.cc:3037] T 3249e60a58ef4fa686ce736e8ebc7c19 P 2bca37447f014aeca45e6117d296bc8e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:11.311658 2283 raft_consensus.cc:3037] T 3249e60a58ef4fa686ce736e8ebc7c19 P bc51e846ab01481e823acbcb5b5a9211 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:11.312337 2283 raft_consensus.cc:2454] T 3249e60a58ef4fa686ce736e8ebc7c19 P bc51e846ab01481e823acbcb5b5a9211 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a17f838c5e94a6d87248e8ca5ea9ce0 in term 1.
I20260619 16:49:11.312433 2410 raft_consensus.cc:2454] T 3249e60a58ef4fa686ce736e8ebc7c19 P 2bca37447f014aeca45e6117d296bc8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a17f838c5e94a6d87248e8ca5ea9ce0 in term 1.
I20260619 16:49:11.312538 2497 leader_election.cc:302] T 3249e60a58ef4fa686ce736e8ebc7c19 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a17f838c5e94a6d87248e8ca5ea9ce0, bc51e846ab01481e823acbcb5b5a9211; no voters:
I20260619 16:49:11.312652 2640 raft_consensus.cc:2781] T 3249e60a58ef4fa686ce736e8ebc7c19 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:11.312795 2640 raft_consensus.cc:686] T 3249e60a58ef4fa686ce736e8ebc7c19 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 1 LEADER]: Becoming Leader. State: Replica: 4a17f838c5e94a6d87248e8ca5ea9ce0, State: Running, Role: LEADER
I20260619 16:49:11.312870 2640 consensus_queue.cc:237] T 3249e60a58ef4fa686ce736e8ebc7c19 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } }
I20260619 16:49:11.313719 2125 catalog_manager.cc:5495] T 3249e60a58ef4fa686ce736e8ebc7c19 P 4a17f838c5e94a6d87248e8ca5ea9ce0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a17f838c5e94a6d87248e8ca5ea9ce0 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } health_report { overall_health: HEALTHY } } }
I20260619 16:49:11.341286 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.338888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52768 (local address 127.0.98.3:32883)
0619 16:49:11.338936 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:11.338938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:11.338950 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:49:11.339035 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:11.339036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:11.339056 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:11.339145 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:11.339271 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.339787 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.340522 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.340779 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.340840 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.340961 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:11.341088 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:11.341144 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:11.341160 (+ 16us) server_negotiation.cc:299] Negotiation successful
0619 16:49:11.341180 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 16:49:11.346942 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.344176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37025 (local address 127.0.98.2:41853)
0619 16:49:11.344246 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:11.344248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:11.344396 (+ 148us) server_negotiation.cc:407] Connection header received
0619 16:49:11.344450 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:11.344451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:11.344464 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:11.344511 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:11.344649 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.345309 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.346143 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.346499 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.346572 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.346732 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:11.346761 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:11.346797 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:11.346840 (+ 43us) server_negotiation.cc:299] Negotiation successful
0619 16:49:11.346857 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 16:49:11.346957 2693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.344034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41853 (local address 127.0.98.3:37025)
0619 16:49:11.344306 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:11.344316 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:11.344399 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:11.344511 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:11.344512 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:11.344514 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:49:11.344595 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.344598 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.345412 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.345413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.346056 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.346058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.346582 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.346583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.346636 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.346654 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:49:11.346804 (+ 150us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:49:11.346805 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 16:49:11.346837 (+ 32us) client_negotiation.cc:240] Negotiation successful
0619 16:49:11.346881 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":153,"threads_started":1}
I20260619 16:49:11.349298 2410 raft_consensus.cc:1261] T 3249e60a58ef4fa686ce736e8ebc7c19 P 2bca37447f014aeca45e6117d296bc8e [term 1 FOLLOWER]: Refusing update from remote peer 4a17f838c5e94a6d87248e8ca5ea9ce0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:11.349370 2282 raft_consensus.cc:1261] T 3249e60a58ef4fa686ce736e8ebc7c19 P bc51e846ab01481e823acbcb5b5a9211 [term 1 FOLLOWER]: Refusing update from remote peer 4a17f838c5e94a6d87248e8ca5ea9ce0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:11.349824 2689 consensus_queue.cc:1035] T 3249e60a58ef4fa686ce736e8ebc7c19 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:11.349980 2640 consensus_queue.cc:1035] T 3249e60a58ef4fa686ce736e8ebc7c19 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:11.357645 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.355194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32883 (local address 127.0.98.2:54727)
0619 16:49:11.355236 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:11.355250 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:11.355290 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:11.355465 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:11.355466 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:11.355467 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:49:11.355519 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.355520 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.356267 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.356268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.356899 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:11.356901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.357238 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.357239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:11.357297 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.357316 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:49:11.357455 (+ 139us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:49:11.357457 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 16:49:11.357504 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 16:49:11.357576 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260619 16:49:11.357626 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:11.355290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54727 (local address 127.0.98.3:32883)
0619 16:49:11.355337 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:11.355339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:11.355347 (+ 8us) server_negotiation.cc:407] Connection header received
0619 16:49:11.355369 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:11.355369 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:11.355381 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:11.355429 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:11.355592 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.356181 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.356973 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:11.357181 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:11.357237 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:11.357370 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:11.357408 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:11.357450 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:11.357508 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 16:49:11.357536 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 16:49:11.358219 2663 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 16:49:12.884958 2125 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:12.885104 2125 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:12.886422 2125 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8990279310d642648682b1df509b9968: Sending DeleteTablet for 3 replicas of tablet 3249e60a58ef4fa686ce736e8ebc7c19
I20260619 16:49:12.886786 2543 tablet_service.cc:1514] Processing DeleteTablet for tablet 3249e60a58ef4fa686ce736e8ebc7c19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52762
I20260619 16:49:12.886806 2263 tablet_service.cc:1514] Processing DeleteTablet for tablet 3249e60a58ef4fa686ce736e8ebc7c19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40674
I20260619 16:49:12.886897 2403 tablet_service.cc:1514] Processing DeleteTablet for tablet 3249e60a58ef4fa686ce736e8ebc7c19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42588
I20260619 16:49:12.887074 2720 tablet_replica.cc:330] T 3249e60a58ef4fa686ce736e8ebc7c19 P 4a17f838c5e94a6d87248e8ca5ea9ce0: stopping tablet replica
I20260619 16:49:12.887133 2722 tablet_replica.cc:330] T 3249e60a58ef4fa686ce736e8ebc7c19 P 2bca37447f014aeca45e6117d296bc8e: stopping tablet replica
I20260619 16:49:12.887315 2722 raft_consensus.cc:2229] T 3249e60a58ef4fa686ce736e8ebc7c19 P 2bca37447f014aeca45e6117d296bc8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:12.887389 2722 raft_consensus.cc:2258] T 3249e60a58ef4fa686ce736e8ebc7c19 P 2bca37447f014aeca45e6117d296bc8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:12.887816 2722 ts_tablet_manager.cc:1905] T 3249e60a58ef4fa686ce736e8ebc7c19 P 2bca37447f014aeca45e6117d296bc8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:12.888077 2720 raft_consensus.cc:2229] T 3249e60a58ef4fa686ce736e8ebc7c19 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:49:12.889858 2720 raft_consensus.cc:2258] T 3249e60a58ef4fa686ce736e8ebc7c19 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:12.890254 2722 ts_tablet_manager.cc:1918] T 3249e60a58ef4fa686ce736e8ebc7c19 P 2bca37447f014aeca45e6117d296bc8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 16:49:12.890318 2722 log.cc:1196] T 3249e60a58ef4fa686ce736e8ebc7c19 P 2bca37447f014aeca45e6117d296bc8e: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-1/wal/wals/3249e60a58ef4fa686ce736e8ebc7c19
I20260619 16:49:12.890532 2722 ts_tablet_manager.cc:1939] T 3249e60a58ef4fa686ce736e8ebc7c19 P 2bca37447f014aeca45e6117d296bc8e: Deleting consensus metadata
I20260619 16:49:12.891238 2111 catalog_manager.cc:4841] TS 2bca37447f014aeca45e6117d296bc8e (127.0.98.2:41853): tablet 3249e60a58ef4fa686ce736e8ebc7c19 (table test-table [id=db40df6aa6e94126a69700458491a5c7]) successfully deleted
I20260619 16:49:12.892737 2720 ts_tablet_manager.cc:1905] T 3249e60a58ef4fa686ce736e8ebc7c19 P 4a17f838c5e94a6d87248e8ca5ea9ce0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:12.894389 2721 tablet_replica.cc:330] T 3249e60a58ef4fa686ce736e8ebc7c19 P bc51e846ab01481e823acbcb5b5a9211: stopping tablet replica
I20260619 16:49:12.894435 2720 ts_tablet_manager.cc:1918] T 3249e60a58ef4fa686ce736e8ebc7c19 P 4a17f838c5e94a6d87248e8ca5ea9ce0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 16:49:12.894493 2720 log.cc:1196] T 3249e60a58ef4fa686ce736e8ebc7c19 P 4a17f838c5e94a6d87248e8ca5ea9ce0: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-2/wal/wals/3249e60a58ef4fa686ce736e8ebc7c19
I20260619 16:49:12.894495 2721 raft_consensus.cc:2229] T 3249e60a58ef4fa686ce736e8ebc7c19 P bc51e846ab01481e823acbcb5b5a9211 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:12.894572 2721 raft_consensus.cc:2258] T 3249e60a58ef4fa686ce736e8ebc7c19 P bc51e846ab01481e823acbcb5b5a9211 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:12.894748 2720 ts_tablet_manager.cc:1939] T 3249e60a58ef4fa686ce736e8ebc7c19 P 4a17f838c5e94a6d87248e8ca5ea9ce0: Deleting consensus metadata
I20260619 16:49:12.894881 2721 ts_tablet_manager.cc:1905] T 3249e60a58ef4fa686ce736e8ebc7c19 P bc51e846ab01481e823acbcb5b5a9211: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:12.895251 2110 catalog_manager.cc:4841] TS 4a17f838c5e94a6d87248e8ca5ea9ce0 (127.0.98.3:32883): tablet 3249e60a58ef4fa686ce736e8ebc7c19 (table test-table [id=db40df6aa6e94126a69700458491a5c7]) successfully deleted
I20260619 16:49:12.897073 2721 ts_tablet_manager.cc:1918] T 3249e60a58ef4fa686ce736e8ebc7c19 P bc51e846ab01481e823acbcb5b5a9211: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 16:49:12.897161 2721 log.cc:1196] T 3249e60a58ef4fa686ce736e8ebc7c19 P bc51e846ab01481e823acbcb5b5a9211: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-0/wal/wals/3249e60a58ef4fa686ce736e8ebc7c19
I20260619 16:49:12.897631 2721 ts_tablet_manager.cc:1939] T 3249e60a58ef4fa686ce736e8ebc7c19 P bc51e846ab01481e823acbcb5b5a9211: Deleting consensus metadata
I20260619 16:49:12.898258 2113 catalog_manager.cc:4841] TS bc51e846ab01481e823acbcb5b5a9211 (127.0.98.1:39321): tablet 3249e60a58ef4fa686ce736e8ebc7c19 (table test-table [id=db40df6aa6e94126a69700458491a5c7]) successfully deleted
Jun 19 16:49:12 dist-test-slave-jsw9 krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887752, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 16:49:12 dist-test-slave-jsw9 krb5kdc[2081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887752, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:12.920361 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:12.906867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40674 (local address 127.0.98.62:46337)
0619 16:49:12.907025 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:12.907027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:12.907037 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:49:12.907065 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:12.907066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:12.907078 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:12.907152 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:12.907920 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:12.908530 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:12.909492 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:12.909786 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:12.909873 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:12.917109 (+ 7236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:12.917112 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:12.917112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:12.917123 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:12.918869 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:12.919325 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:12.919326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:12.919327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:12.919396 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:12.919682 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:12.919683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:12.919684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:12.919873 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:12.919976 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:12.920154 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 16:49:12.920254 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":91,"threads_started":1}
I20260619 16:49:12.922366 2125 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:49:12.922556 2125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:49:12.924824 2543 tablet_service.cc:1467] Processing CreateTablet for tablet ed0ee731ab4a4c70bd2ac3840abaf636 (DEFAULT_TABLE table=test-table [id=582be998bdfe499f986cf6574363c6c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:12.924803 2263 tablet_service.cc:1467] Processing CreateTablet for tablet ed0ee731ab4a4c70bd2ac3840abaf636 (DEFAULT_TABLE table=test-table [id=582be998bdfe499f986cf6574363c6c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:12.924834 2403 tablet_service.cc:1467] Processing CreateTablet for tablet ed0ee731ab4a4c70bd2ac3840abaf636 (DEFAULT_TABLE table=test-table [id=582be998bdfe499f986cf6574363c6c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:12.924988 2543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed0ee731ab4a4c70bd2ac3840abaf636. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:12.924989 2263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed0ee731ab4a4c70bd2ac3840abaf636. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:12.924988 2403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed0ee731ab4a4c70bd2ac3840abaf636. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:12.926808 2731 tablet_bootstrap.cc:492] T ed0ee731ab4a4c70bd2ac3840abaf636 P 2bca37447f014aeca45e6117d296bc8e: Bootstrap starting.
I20260619 16:49:12.927140 2732 tablet_bootstrap.cc:492] T ed0ee731ab4a4c70bd2ac3840abaf636 P bc51e846ab01481e823acbcb5b5a9211: Bootstrap starting.
I20260619 16:49:12.927366 2731 tablet_bootstrap.cc:654] T ed0ee731ab4a4c70bd2ac3840abaf636 P 2bca37447f014aeca45e6117d296bc8e: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:12.927572 2732 tablet_bootstrap.cc:654] T ed0ee731ab4a4c70bd2ac3840abaf636 P bc51e846ab01481e823acbcb5b5a9211: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:12.927805 2733 tablet_bootstrap.cc:492] T ed0ee731ab4a4c70bd2ac3840abaf636 P 4a17f838c5e94a6d87248e8ca5ea9ce0: Bootstrap starting.
I20260619 16:49:12.928149 2731 tablet_bootstrap.cc:492] T ed0ee731ab4a4c70bd2ac3840abaf636 P 2bca37447f014aeca45e6117d296bc8e: No bootstrap required, opened a new log
I20260619 16:49:12.928226 2731 ts_tablet_manager.cc:1397] T ed0ee731ab4a4c70bd2ac3840abaf636 P 2bca37447f014aeca45e6117d296bc8e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:12.928429 2733 tablet_bootstrap.cc:654] T ed0ee731ab4a4c70bd2ac3840abaf636 P 4a17f838c5e94a6d87248e8ca5ea9ce0: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:12.928421 2731 raft_consensus.cc:348] T ed0ee731ab4a4c70bd2ac3840abaf636 P 2bca37447f014aeca45e6117d296bc8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } }
I20260619 16:49:12.928522 2732 tablet_bootstrap.cc:492] T ed0ee731ab4a4c70bd2ac3840abaf636 P bc51e846ab01481e823acbcb5b5a9211: No bootstrap required, opened a new log
I20260619 16:49:12.928540 2731 raft_consensus.cc:374] T ed0ee731ab4a4c70bd2ac3840abaf636 P 2bca37447f014aeca45e6117d296bc8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:12.928571 2731 raft_consensus.cc:729] T ed0ee731ab4a4c70bd2ac3840abaf636 P 2bca37447f014aeca45e6117d296bc8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bca37447f014aeca45e6117d296bc8e, State: Initialized, Role: FOLLOWER
I20260619 16:49:12.928572 2732 ts_tablet_manager.cc:1397] T ed0ee731ab4a4c70bd2ac3840abaf636 P bc51e846ab01481e823acbcb5b5a9211: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:12.928639 2731 consensus_queue.cc:260] T ed0ee731ab4a4c70bd2ac3840abaf636 P 2bca37447f014aeca45e6117d296bc8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } }
I20260619 16:49:12.928731 2732 raft_consensus.cc:348] T ed0ee731ab4a4c70bd2ac3840abaf636 P bc51e846ab01481e823acbcb5b5a9211 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } }
I20260619 16:49:12.928781 2731 ts_tablet_manager.cc:1428] T ed0ee731ab4a4c70bd2ac3840abaf636 P 2bca37447f014aeca45e6117d296bc8e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:12.928807 2732 raft_consensus.cc:374] T ed0ee731ab4a4c70bd2ac3840abaf636 P bc51e846ab01481e823acbcb5b5a9211 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:12.928825 2732 raft_consensus.cc:729] T ed0ee731ab4a4c70bd2ac3840abaf636 P bc51e846ab01481e823acbcb5b5a9211 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc51e846ab01481e823acbcb5b5a9211, State: Initialized, Role: FOLLOWER
I20260619 16:49:12.928905 2732 consensus_queue.cc:260] T ed0ee731ab4a4c70bd2ac3840abaf636 P bc51e846ab01481e823acbcb5b5a9211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } }
I20260619 16:49:12.929107 2732 ts_tablet_manager.cc:1428] T ed0ee731ab4a4c70bd2ac3840abaf636 P bc51e846ab01481e823acbcb5b5a9211: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:12.929659 2733 tablet_bootstrap.cc:492] T ed0ee731ab4a4c70bd2ac3840abaf636 P 4a17f838c5e94a6d87248e8ca5ea9ce0: No bootstrap required, opened a new log
I20260619 16:49:12.929782 2733 ts_tablet_manager.cc:1397] T ed0ee731ab4a4c70bd2ac3840abaf636 P 4a17f838c5e94a6d87248e8ca5ea9ce0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:49:12.929965 2733 raft_consensus.cc:348] T ed0ee731ab4a4c70bd2ac3840abaf636 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } }
I20260619 16:49:12.930047 2733 raft_consensus.cc:374] T ed0ee731ab4a4c70bd2ac3840abaf636 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:12.930068 2733 raft_consensus.cc:729] T ed0ee731ab4a4c70bd2ac3840abaf636 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a17f838c5e94a6d87248e8ca5ea9ce0, State: Initialized, Role: FOLLOWER
I20260619 16:49:12.930120 2733 consensus_queue.cc:260] T ed0ee731ab4a4c70bd2ac3840abaf636 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } }
I20260619 16:49:12.930303 2733 ts_tablet_manager.cc:1428] T ed0ee731ab4a4c70bd2ac3840abaf636 P 4a17f838c5e94a6d87248e8ca5ea9ce0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:12.943462 2737 raft_consensus.cc:482] T ed0ee731ab4a4c70bd2ac3840abaf636 P bc51e846ab01481e823acbcb5b5a9211 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:12.943615 2737 raft_consensus.cc:504] T ed0ee731ab4a4c70bd2ac3840abaf636 P bc51e846ab01481e823acbcb5b5a9211 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } }
I20260619 16:49:12.943856 2737 leader_election.cc:288] T ed0ee731ab4a4c70bd2ac3840abaf636 P bc51e846ab01481e823acbcb5b5a9211 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2bca37447f014aeca45e6117d296bc8e (127.0.98.2:41853), 4a17f838c5e94a6d87248e8ca5ea9ce0 (127.0.98.3:32883)
I20260619 16:49:12.944098 2410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed0ee731ab4a4c70bd2ac3840abaf636" candidate_uuid: "bc51e846ab01481e823acbcb5b5a9211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bca37447f014aeca45e6117d296bc8e" is_pre_election: true
I20260619 16:49:12.944106 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed0ee731ab4a4c70bd2ac3840abaf636" candidate_uuid: "bc51e846ab01481e823acbcb5b5a9211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" is_pre_election: true
I20260619 16:49:12.944202 2410 raft_consensus.cc:2454] T ed0ee731ab4a4c70bd2ac3840abaf636 P 2bca37447f014aeca45e6117d296bc8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc51e846ab01481e823acbcb5b5a9211 in term 0.
I20260619 16:49:12.944229 2563 raft_consensus.cc:2454] T ed0ee731ab4a4c70bd2ac3840abaf636 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc51e846ab01481e823acbcb5b5a9211 in term 0.
I20260619 16:49:12.944454 2215 leader_election.cc:302] T ed0ee731ab4a4c70bd2ac3840abaf636 P bc51e846ab01481e823acbcb5b5a9211 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bca37447f014aeca45e6117d296bc8e, bc51e846ab01481e823acbcb5b5a9211; no voters:
I20260619 16:49:12.944595 2737 raft_consensus.cc:2781] T ed0ee731ab4a4c70bd2ac3840abaf636 P bc51e846ab01481e823acbcb5b5a9211 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:49:12.944660 2737 raft_consensus.cc:482] T ed0ee731ab4a4c70bd2ac3840abaf636 P bc51e846ab01481e823acbcb5b5a9211 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:49:12.944680 2737 raft_consensus.cc:3037] T ed0ee731ab4a4c70bd2ac3840abaf636 P bc51e846ab01481e823acbcb5b5a9211 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:12.945725 2737 raft_consensus.cc:504] T ed0ee731ab4a4c70bd2ac3840abaf636 P bc51e846ab01481e823acbcb5b5a9211 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } }
I20260619 16:49:12.945891 2737 leader_election.cc:288] T ed0ee731ab4a4c70bd2ac3840abaf636 P bc51e846ab01481e823acbcb5b5a9211 [CANDIDATE]: Term 1 election: Requested vote from peers 2bca37447f014aeca45e6117d296bc8e (127.0.98.2:41853), 4a17f838c5e94a6d87248e8ca5ea9ce0 (127.0.98.3:32883)
I20260619 16:49:12.946236 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed0ee731ab4a4c70bd2ac3840abaf636" candidate_uuid: "bc51e846ab01481e823acbcb5b5a9211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0"
I20260619 16:49:12.946336 2563 raft_consensus.cc:3037] T ed0ee731ab4a4c70bd2ac3840abaf636 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:12.946336 2410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed0ee731ab4a4c70bd2ac3840abaf636" candidate_uuid: "bc51e846ab01481e823acbcb5b5a9211" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bca37447f014aeca45e6117d296bc8e"
I20260619 16:49:12.946501 2410 raft_consensus.cc:3037] T ed0ee731ab4a4c70bd2ac3840abaf636 P 2bca37447f014aeca45e6117d296bc8e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:12.947325 2410 raft_consensus.cc:2454] T ed0ee731ab4a4c70bd2ac3840abaf636 P 2bca37447f014aeca45e6117d296bc8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc51e846ab01481e823acbcb5b5a9211 in term 1.
I20260619 16:49:12.947566 2215 leader_election.cc:302] T ed0ee731ab4a4c70bd2ac3840abaf636 P bc51e846ab01481e823acbcb5b5a9211 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bca37447f014aeca45e6117d296bc8e, bc51e846ab01481e823acbcb5b5a9211; no voters:
I20260619 16:49:12.947669 2737 raft_consensus.cc:2781] T ed0ee731ab4a4c70bd2ac3840abaf636 P bc51e846ab01481e823acbcb5b5a9211 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:12.947847 2563 raft_consensus.cc:2454] T ed0ee731ab4a4c70bd2ac3840abaf636 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc51e846ab01481e823acbcb5b5a9211 in term 1.
I20260619 16:49:12.947862 2737 raft_consensus.cc:686] T ed0ee731ab4a4c70bd2ac3840abaf636 P bc51e846ab01481e823acbcb5b5a9211 [term 1 LEADER]: Becoming Leader. State: Replica: bc51e846ab01481e823acbcb5b5a9211, State: Running, Role: LEADER
I20260619 16:49:12.947937 2737 consensus_queue.cc:237] T ed0ee731ab4a4c70bd2ac3840abaf636 P bc51e846ab01481e823acbcb5b5a9211 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } }
I20260619 16:49:12.948853 2125 catalog_manager.cc:5495] T ed0ee731ab4a4c70bd2ac3840abaf636 P bc51e846ab01481e823acbcb5b5a9211 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc51e846ab01481e823acbcb5b5a9211 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "bc51e846ab01481e823acbcb5b5a9211" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } health_report { overall_health: UNKNOWN } } }
I20260619 16:49:12.962767 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:12.959881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40694 (local address 127.0.98.1:39321)
0619 16:49:12.960072 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:12.960073 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:12.960080 (+ 7us) server_negotiation.cc:407] Connection header received
0619 16:49:12.960103 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:12.960103 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:12.960115 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:12.960180 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:12.960333 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:12.960878 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:12.961980 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:12.962259 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:12.962331 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:12.962486 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:12.962537 (+ 51us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:12.962582 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:12.962633 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 16:49:12.962661 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":110,"threads_started":1}
I20260619 16:49:12.966522 2563 raft_consensus.cc:1261] T ed0ee731ab4a4c70bd2ac3840abaf636 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 1 FOLLOWER]: Refusing update from remote peer bc51e846ab01481e823acbcb5b5a9211: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:12.966476 2410 raft_consensus.cc:1261] T ed0ee731ab4a4c70bd2ac3840abaf636 P 2bca37447f014aeca45e6117d296bc8e [term 1 FOLLOWER]: Refusing update from remote peer bc51e846ab01481e823acbcb5b5a9211: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:12.966869 2737 consensus_queue.cc:1035] T ed0ee731ab4a4c70bd2ac3840abaf636 P bc51e846ab01481e823acbcb5b5a9211 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:12.967060 2738 consensus_queue.cc:1035] T ed0ee731ab4a4c70bd2ac3840abaf636 P bc51e846ab01481e823acbcb5b5a9211 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:12.969862 2702 mvcc.cc:205] Tried to move back new op lower bound from 7298612236148584448 to 7298612236075524096. Current Snapshot: MvccSnapshot[applied={T|T < 7298612236148584448}]
I20260619 16:49:12.985628 2125 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:12.985796 2125 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:12.986951 2125 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8990279310d642648682b1df509b9968: Sending DeleteTablet for 3 replicas of tablet ed0ee731ab4a4c70bd2ac3840abaf636
I20260619 16:49:12.987607 2263 tablet_service.cc:1514] Processing DeleteTablet for tablet ed0ee731ab4a4c70bd2ac3840abaf636 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40674
I20260619 16:49:12.987622 2403 tablet_service.cc:1514] Processing DeleteTablet for tablet ed0ee731ab4a4c70bd2ac3840abaf636 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42588
I20260619 16:49:12.987756 2721 tablet_replica.cc:330] T ed0ee731ab4a4c70bd2ac3840abaf636 P bc51e846ab01481e823acbcb5b5a9211: stopping tablet replica
I20260619 16:49:12.987762 2722 tablet_replica.cc:330] T ed0ee731ab4a4c70bd2ac3840abaf636 P 2bca37447f014aeca45e6117d296bc8e: stopping tablet replica
I20260619 16:49:12.987773 2543 tablet_service.cc:1514] Processing DeleteTablet for tablet ed0ee731ab4a4c70bd2ac3840abaf636 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52762
I20260619 16:49:12.987828 2722 raft_consensus.cc:2229] T ed0ee731ab4a4c70bd2ac3840abaf636 P 2bca37447f014aeca45e6117d296bc8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:12.987825 2721 raft_consensus.cc:2229] T ed0ee731ab4a4c70bd2ac3840abaf636 P bc51e846ab01481e823acbcb5b5a9211 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:49:12.987874 2720 tablet_replica.cc:330] T ed0ee731ab4a4c70bd2ac3840abaf636 P 4a17f838c5e94a6d87248e8ca5ea9ce0: stopping tablet replica
I20260619 16:49:12.987888 2722 raft_consensus.cc:2258] T ed0ee731ab4a4c70bd2ac3840abaf636 P 2bca37447f014aeca45e6117d296bc8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:12.987921 2720 raft_consensus.cc:2229] T ed0ee731ab4a4c70bd2ac3840abaf636 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:12.987934 2721 raft_consensus.cc:2258] T ed0ee731ab4a4c70bd2ac3840abaf636 P bc51e846ab01481e823acbcb5b5a9211 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:12.987962 2720 raft_consensus.cc:2258] T ed0ee731ab4a4c70bd2ac3840abaf636 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:12.988444 2722 ts_tablet_manager.cc:1905] T ed0ee731ab4a4c70bd2ac3840abaf636 P 2bca37447f014aeca45e6117d296bc8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:12.988315 2125 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 16:49:12.988415 2721 ts_tablet_manager.cc:1905] T ed0ee731ab4a4c70bd2ac3840abaf636 P bc51e846ab01481e823acbcb5b5a9211: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 16:49:12.988582 2125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:49:12.990650 2720 ts_tablet_manager.cc:1905] T ed0ee731ab4a4c70bd2ac3840abaf636 P 4a17f838c5e94a6d87248e8ca5ea9ce0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:12.991009 2722 ts_tablet_manager.cc:1918] T ed0ee731ab4a4c70bd2ac3840abaf636 P 2bca37447f014aeca45e6117d296bc8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:49:12.991071 2722 log.cc:1196] T ed0ee731ab4a4c70bd2ac3840abaf636 P 2bca37447f014aeca45e6117d296bc8e: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-1/wal/wals/ed0ee731ab4a4c70bd2ac3840abaf636
I20260619 16:49:12.991120 2721 ts_tablet_manager.cc:1918] T ed0ee731ab4a4c70bd2ac3840abaf636 P bc51e846ab01481e823acbcb5b5a9211: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:49:12.991199 2721 log.cc:1196] T ed0ee731ab4a4c70bd2ac3840abaf636 P bc51e846ab01481e823acbcb5b5a9211: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-0/wal/wals/ed0ee731ab4a4c70bd2ac3840abaf636
I20260619 16:49:12.991361 2721 ts_tablet_manager.cc:1939] T ed0ee731ab4a4c70bd2ac3840abaf636 P bc51e846ab01481e823acbcb5b5a9211: Deleting consensus metadata
I20260619 16:49:12.991477 2722 ts_tablet_manager.cc:1939] T ed0ee731ab4a4c70bd2ac3840abaf636 P 2bca37447f014aeca45e6117d296bc8e: Deleting consensus metadata
I20260619 16:49:12.992065 2113 catalog_manager.cc:4841] TS bc51e846ab01481e823acbcb5b5a9211 (127.0.98.1:39321): tablet ed0ee731ab4a4c70bd2ac3840abaf636 (table test-table [id=582be998bdfe499f986cf6574363c6c0]) successfully deleted
I20260619 16:49:12.992101 2111 catalog_manager.cc:4841] TS 2bca37447f014aeca45e6117d296bc8e (127.0.98.2:41853): tablet ed0ee731ab4a4c70bd2ac3840abaf636 (table test-table [id=582be998bdfe499f986cf6574363c6c0]) successfully deleted
I20260619 16:49:12.992624 2543 tablet_service.cc:1467] Processing CreateTablet for tablet 3597009a8b5d475bb8d31c7cf92b7f66 (DEFAULT_TABLE table=test-table [id=12c00f0014534c18ba14e48bf633d740]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:12.992624 2403 tablet_service.cc:1467] Processing CreateTablet for tablet 3597009a8b5d475bb8d31c7cf92b7f66 (DEFAULT_TABLE table=test-table [id=12c00f0014534c18ba14e48bf633d740]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:12.992624 2263 tablet_service.cc:1467] Processing CreateTablet for tablet 3597009a8b5d475bb8d31c7cf92b7f66 (DEFAULT_TABLE table=test-table [id=12c00f0014534c18ba14e48bf633d740]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:12.992750 2403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3597009a8b5d475bb8d31c7cf92b7f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:12.992750 2263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3597009a8b5d475bb8d31c7cf92b7f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:12.992753 2543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3597009a8b5d475bb8d31c7cf92b7f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:12.993136 2720 ts_tablet_manager.cc:1918] T ed0ee731ab4a4c70bd2ac3840abaf636 P 4a17f838c5e94a6d87248e8ca5ea9ce0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:49:12.993188 2720 log.cc:1196] T ed0ee731ab4a4c70bd2ac3840abaf636 P 4a17f838c5e94a6d87248e8ca5ea9ce0: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-2/wal/wals/ed0ee731ab4a4c70bd2ac3840abaf636
I20260619 16:49:12.993378 2720 ts_tablet_manager.cc:1939] T ed0ee731ab4a4c70bd2ac3840abaf636 P 4a17f838c5e94a6d87248e8ca5ea9ce0: Deleting consensus metadata
I20260619 16:49:12.994323 2731 tablet_bootstrap.cc:492] T 3597009a8b5d475bb8d31c7cf92b7f66 P 2bca37447f014aeca45e6117d296bc8e: Bootstrap starting.
I20260619 16:49:12.994721 2731 tablet_bootstrap.cc:654] T 3597009a8b5d475bb8d31c7cf92b7f66 P 2bca37447f014aeca45e6117d296bc8e: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:12.995000 2733 tablet_bootstrap.cc:492] T 3597009a8b5d475bb8d31c7cf92b7f66 P 4a17f838c5e94a6d87248e8ca5ea9ce0: Bootstrap starting.
I20260619 16:49:12.995148 2732 tablet_bootstrap.cc:492] T 3597009a8b5d475bb8d31c7cf92b7f66 P bc51e846ab01481e823acbcb5b5a9211: Bootstrap starting.
I20260619 16:49:12.995175 2110 catalog_manager.cc:4841] TS 4a17f838c5e94a6d87248e8ca5ea9ce0 (127.0.98.3:32883): tablet ed0ee731ab4a4c70bd2ac3840abaf636 (table test-table [id=582be998bdfe499f986cf6574363c6c0]) successfully deleted
I20260619 16:49:12.995352 2733 tablet_bootstrap.cc:654] T 3597009a8b5d475bb8d31c7cf92b7f66 P 4a17f838c5e94a6d87248e8ca5ea9ce0: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:12.995538 2732 tablet_bootstrap.cc:654] T 3597009a8b5d475bb8d31c7cf92b7f66 P bc51e846ab01481e823acbcb5b5a9211: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:12.995756 2731 tablet_bootstrap.cc:492] T 3597009a8b5d475bb8d31c7cf92b7f66 P 2bca37447f014aeca45e6117d296bc8e: No bootstrap required, opened a new log
I20260619 16:49:12.995819 2731 ts_tablet_manager.cc:1397] T 3597009a8b5d475bb8d31c7cf92b7f66 P 2bca37447f014aeca45e6117d296bc8e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:49:12.995967 2731 raft_consensus.cc:348] T 3597009a8b5d475bb8d31c7cf92b7f66 P 2bca37447f014aeca45e6117d296bc8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } }
I20260619 16:49:12.996050 2731 raft_consensus.cc:374] T 3597009a8b5d475bb8d31c7cf92b7f66 P 2bca37447f014aeca45e6117d296bc8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:12.996071 2731 raft_consensus.cc:729] T 3597009a8b5d475bb8d31c7cf92b7f66 P 2bca37447f014aeca45e6117d296bc8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bca37447f014aeca45e6117d296bc8e, State: Initialized, Role: FOLLOWER
I20260619 16:49:12.996125 2733 tablet_bootstrap.cc:492] T 3597009a8b5d475bb8d31c7cf92b7f66 P 4a17f838c5e94a6d87248e8ca5ea9ce0: No bootstrap required, opened a new log
I20260619 16:49:12.996120 2731 consensus_queue.cc:260] T 3597009a8b5d475bb8d31c7cf92b7f66 P 2bca37447f014aeca45e6117d296bc8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } }
I20260619 16:49:12.996176 2733 ts_tablet_manager.cc:1397] T 3597009a8b5d475bb8d31c7cf92b7f66 P 4a17f838c5e94a6d87248e8ca5ea9ce0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 16:49:12.996219 2731 ts_tablet_manager.cc:1428] T 3597009a8b5d475bb8d31c7cf92b7f66 P 2bca37447f014aeca45e6117d296bc8e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:12.996263 2732 tablet_bootstrap.cc:492] T 3597009a8b5d475bb8d31c7cf92b7f66 P bc51e846ab01481e823acbcb5b5a9211: No bootstrap required, opened a new log
I20260619 16:49:12.996305 2732 ts_tablet_manager.cc:1397] T 3597009a8b5d475bb8d31c7cf92b7f66 P bc51e846ab01481e823acbcb5b5a9211: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:12.996310 2733 raft_consensus.cc:348] T 3597009a8b5d475bb8d31c7cf92b7f66 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } }
I20260619 16:49:12.996382 2733 raft_consensus.cc:374] T 3597009a8b5d475bb8d31c7cf92b7f66 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:12.996400 2733 raft_consensus.cc:729] T 3597009a8b5d475bb8d31c7cf92b7f66 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a17f838c5e94a6d87248e8ca5ea9ce0, State: Initialized, Role: FOLLOWER
I20260619 16:49:12.996441 2733 consensus_queue.cc:260] T 3597009a8b5d475bb8d31c7cf92b7f66 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } }
I20260619 16:49:12.996447 2732 raft_consensus.cc:348] T 3597009a8b5d475bb8d31c7cf92b7f66 P bc51e846ab01481e823acbcb5b5a9211 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } }
I20260619 16:49:12.996513 2732 raft_consensus.cc:374] T 3597009a8b5d475bb8d31c7cf92b7f66 P bc51e846ab01481e823acbcb5b5a9211 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:12.996531 2732 raft_consensus.cc:729] T 3597009a8b5d475bb8d31c7cf92b7f66 P bc51e846ab01481e823acbcb5b5a9211 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc51e846ab01481e823acbcb5b5a9211, State: Initialized, Role: FOLLOWER
I20260619 16:49:12.996533 2733 ts_tablet_manager.cc:1428] T 3597009a8b5d475bb8d31c7cf92b7f66 P 4a17f838c5e94a6d87248e8ca5ea9ce0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:12.996574 2732 consensus_queue.cc:260] T 3597009a8b5d475bb8d31c7cf92b7f66 P bc51e846ab01481e823acbcb5b5a9211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } }
I20260619 16:49:12.996665 2732 ts_tablet_manager.cc:1428] T 3597009a8b5d475bb8d31c7cf92b7f66 P bc51e846ab01481e823acbcb5b5a9211: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 16:49:13.147938 2701 raft_consensus.cc:482] T 3597009a8b5d475bb8d31c7cf92b7f66 P 2bca37447f014aeca45e6117d296bc8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:13.148073 2701 raft_consensus.cc:504] T 3597009a8b5d475bb8d31c7cf92b7f66 P 2bca37447f014aeca45e6117d296bc8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } }
I20260619 16:49:13.148289 2701 leader_election.cc:288] T 3597009a8b5d475bb8d31c7cf92b7f66 P 2bca37447f014aeca45e6117d296bc8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc51e846ab01481e823acbcb5b5a9211 (127.0.98.1:39321), 4a17f838c5e94a6d87248e8ca5ea9ce0 (127.0.98.3:32883)
I20260619 16:49:13.148574 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3597009a8b5d475bb8d31c7cf92b7f66" candidate_uuid: "2bca37447f014aeca45e6117d296bc8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" is_pre_election: true
I20260619 16:49:13.148579 2282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3597009a8b5d475bb8d31c7cf92b7f66" candidate_uuid: "2bca37447f014aeca45e6117d296bc8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc51e846ab01481e823acbcb5b5a9211" is_pre_election: true
I20260619 16:49:13.148676 2563 raft_consensus.cc:2454] T 3597009a8b5d475bb8d31c7cf92b7f66 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bca37447f014aeca45e6117d296bc8e in term 0.
I20260619 16:49:13.148689 2282 raft_consensus.cc:2454] T 3597009a8b5d475bb8d31c7cf92b7f66 P bc51e846ab01481e823acbcb5b5a9211 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bca37447f014aeca45e6117d296bc8e in term 0.
I20260619 16:49:13.148895 2357 leader_election.cc:302] T 3597009a8b5d475bb8d31c7cf92b7f66 P 2bca37447f014aeca45e6117d296bc8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bca37447f014aeca45e6117d296bc8e, bc51e846ab01481e823acbcb5b5a9211; no voters:
I20260619 16:49:13.149041 2701 raft_consensus.cc:2781] T 3597009a8b5d475bb8d31c7cf92b7f66 P 2bca37447f014aeca45e6117d296bc8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:49:13.149097 2701 raft_consensus.cc:482] T 3597009a8b5d475bb8d31c7cf92b7f66 P 2bca37447f014aeca45e6117d296bc8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:49:13.149119 2701 raft_consensus.cc:3037] T 3597009a8b5d475bb8d31c7cf92b7f66 P 2bca37447f014aeca45e6117d296bc8e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:13.150094 2701 raft_consensus.cc:504] T 3597009a8b5d475bb8d31c7cf92b7f66 P 2bca37447f014aeca45e6117d296bc8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } }
I20260619 16:49:13.150372 2701 leader_election.cc:288] T 3597009a8b5d475bb8d31c7cf92b7f66 P 2bca37447f014aeca45e6117d296bc8e [CANDIDATE]: Term 1 election: Requested vote from peers bc51e846ab01481e823acbcb5b5a9211 (127.0.98.1:39321), 4a17f838c5e94a6d87248e8ca5ea9ce0 (127.0.98.3:32883)
I20260619 16:49:13.150553 2282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3597009a8b5d475bb8d31c7cf92b7f66" candidate_uuid: "2bca37447f014aeca45e6117d296bc8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc51e846ab01481e823acbcb5b5a9211"
I20260619 16:49:13.150568 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3597009a8b5d475bb8d31c7cf92b7f66" candidate_uuid: "2bca37447f014aeca45e6117d296bc8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0"
I20260619 16:49:13.150646 2563 raft_consensus.cc:3037] T 3597009a8b5d475bb8d31c7cf92b7f66 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:13.150648 2282 raft_consensus.cc:3037] T 3597009a8b5d475bb8d31c7cf92b7f66 P bc51e846ab01481e823acbcb5b5a9211 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:13.151437 2282 raft_consensus.cc:2454] T 3597009a8b5d475bb8d31c7cf92b7f66 P bc51e846ab01481e823acbcb5b5a9211 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bca37447f014aeca45e6117d296bc8e in term 1.
I20260619 16:49:13.151473 2563 raft_consensus.cc:2454] T 3597009a8b5d475bb8d31c7cf92b7f66 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bca37447f014aeca45e6117d296bc8e in term 1.
I20260619 16:49:13.151670 2354 leader_election.cc:302] T 3597009a8b5d475bb8d31c7cf92b7f66 P 2bca37447f014aeca45e6117d296bc8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bca37447f014aeca45e6117d296bc8e, 4a17f838c5e94a6d87248e8ca5ea9ce0; no voters:
I20260619 16:49:13.151782 2701 raft_consensus.cc:2781] T 3597009a8b5d475bb8d31c7cf92b7f66 P 2bca37447f014aeca45e6117d296bc8e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:13.151849 2701 raft_consensus.cc:686] T 3597009a8b5d475bb8d31c7cf92b7f66 P 2bca37447f014aeca45e6117d296bc8e [term 1 LEADER]: Becoming Leader. State: Replica: 2bca37447f014aeca45e6117d296bc8e, State: Running, Role: LEADER
I20260619 16:49:13.151911 2701 consensus_queue.cc:237] T 3597009a8b5d475bb8d31c7cf92b7f66 P 2bca37447f014aeca45e6117d296bc8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } }
I20260619 16:49:13.152673 2124 catalog_manager.cc:5495] T 3597009a8b5d475bb8d31c7cf92b7f66 P 2bca37447f014aeca45e6117d296bc8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2bca37447f014aeca45e6117d296bc8e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2bca37447f014aeca45e6117d296bc8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bca37447f014aeca45e6117d296bc8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 } health_report { overall_health: UNKNOWN } } }
I20260619 16:49:13.189231 2751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:13.186490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42606 (local address 127.0.98.2:41853)
0619 16:49:13.186712 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:13.186714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:13.186722 (+ 8us) server_negotiation.cc:407] Connection header received
0619 16:49:13.186742 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:13.186742 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:13.186755 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:13.186823 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:13.186948 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:13.187674 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:13.188487 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:13.188713 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:13.188781 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:13.188957 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:13.188994 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:13.189042 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:13.189080 (+ 38us) server_negotiation.cc:299] Negotiation successful
0619 16:49:13.189112 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":117,"threads_started":1}
I20260619 16:49:13.194567 2751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:13.192141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41285 (local address 127.0.98.2:41853)
0619 16:49:13.192212 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:13.192214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:13.192282 (+ 68us) server_negotiation.cc:407] Connection header received
0619 16:49:13.192342 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:13.192342 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:13.192352 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:13.192401 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:13.192556 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:13.193110 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:13.193918 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:13.194106 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:13.194163 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:13.194341 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:13.194369 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:13.194410 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:13.194455 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 16:49:13.194476 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 16:49:13.194576 2754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:13.191965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41853 (local address 127.0.98.2:41285)
0619 16:49:13.192214 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:13.192233 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:13.192298 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:13.192410 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:13.192411 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:13.192413 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:49:13.192473 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:13.192475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:13.193192 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:13.193193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:13.193830 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:13.193832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:13.194176 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:13.194177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:13.194269 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:13.194275 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:49:13.194418 (+ 143us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:49:13.194419 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 16:49:13.194454 (+ 35us) client_negotiation.cc:240] Negotiation successful
0619 16:49:13.194498 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":104,"threads_started":1}
I20260619 16:49:13.196977 2563 raft_consensus.cc:1261] T 3597009a8b5d475bb8d31c7cf92b7f66 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 1 FOLLOWER]: Refusing update from remote peer 2bca37447f014aeca45e6117d296bc8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:13.196977 2282 raft_consensus.cc:1261] T 3597009a8b5d475bb8d31c7cf92b7f66 P bc51e846ab01481e823acbcb5b5a9211 [term 1 FOLLOWER]: Refusing update from remote peer 2bca37447f014aeca45e6117d296bc8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:13.197319 2701 consensus_queue.cc:1035] T 3597009a8b5d475bb8d31c7cf92b7f66 P 2bca37447f014aeca45e6117d296bc8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a17f838c5e94a6d87248e8ca5ea9ce0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:13.197461 2698 consensus_queue.cc:1035] T 3597009a8b5d475bb8d31c7cf92b7f66 P 2bca37447f014aeca45e6117d296bc8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc51e846ab01481e823acbcb5b5a9211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 16:49:13.202534 2699 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 16:49:14.726600 2126 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:14.726745 2126 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:14.727996 2126 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8990279310d642648682b1df509b9968: Sending DeleteTablet for 3 replicas of tablet 3597009a8b5d475bb8d31c7cf92b7f66
I20260619 16:49:14.728386 2543 tablet_service.cc:1514] Processing DeleteTablet for tablet 3597009a8b5d475bb8d31c7cf92b7f66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52762
I20260619 16:49:14.728379 2263 tablet_service.cc:1514] Processing DeleteTablet for tablet 3597009a8b5d475bb8d31c7cf92b7f66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40674
I20260619 16:49:14.728485 2402 tablet_service.cc:1514] Processing DeleteTablet for tablet 3597009a8b5d475bb8d31c7cf92b7f66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42588
I20260619 16:49:14.728667 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 2198
I20260619 16:49:14.729867 2779 tablet_replica.cc:330] T 3597009a8b5d475bb8d31c7cf92b7f66 P 2bca37447f014aeca45e6117d296bc8e: stopping tablet replica
I20260619 16:49:14.729964 2779 raft_consensus.cc:2229] T 3597009a8b5d475bb8d31c7cf92b7f66 P 2bca37447f014aeca45e6117d296bc8e [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:49:14.730121 2779 raft_consensus.cc:2258] T 3597009a8b5d475bb8d31c7cf92b7f66 P 2bca37447f014aeca45e6117d296bc8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:14.730501 2777 tablet_replica.cc:330] T 3597009a8b5d475bb8d31c7cf92b7f66 P 4a17f838c5e94a6d87248e8ca5ea9ce0: stopping tablet replica
I20260619 16:49:14.730587 2777 raft_consensus.cc:2229] T 3597009a8b5d475bb8d31c7cf92b7f66 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:14.730633 2777 raft_consensus.cc:2258] T 3597009a8b5d475bb8d31c7cf92b7f66 P 4a17f838c5e94a6d87248e8ca5ea9ce0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:14.731011 2777 ts_tablet_manager.cc:1905] T 3597009a8b5d475bb8d31c7cf92b7f66 P 4a17f838c5e94a6d87248e8ca5ea9ce0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:14.733366 2777 ts_tablet_manager.cc:1918] T 3597009a8b5d475bb8d31c7cf92b7f66 P 4a17f838c5e94a6d87248e8ca5ea9ce0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 16:49:14.733456 2777 log.cc:1196] T 3597009a8b5d475bb8d31c7cf92b7f66 P 4a17f838c5e94a6d87248e8ca5ea9ce0: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781887741903642-392-0/minicluster-data/ts-2/wal/wals/3597009a8b5d475bb8d31c7cf92b7f66
I20260619 16:49:14.733680 2777 ts_tablet_manager.cc:1939] T 3597009a8b5d475bb8d31c7cf92b7f66 P 4a17f838c5e94a6d87248e8ca5ea9ce0: Deleting consensus metadata
I20260619 16:49:14.734642 2779 ts_tablet_manager.cc:1905] T 3597009a8b5d475bb8d31c7cf92b7f66 P 2bca37447f014aeca45e6117d296bc8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:14.734670 2110 catalog_manager.cc:4841] TS 4a17f838c5e94a6d87248e8ca5ea9ce0 (127.0.98.3:32883): tablet 3597009a8b5d475bb8d31c7cf92b7f66 (table test-table [id=12c00f0014534c18ba14e48bf633d740]) successfully deleted
W20260619 16:49:14.736296 2113 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39321 (error 108)
I20260619 16:49:14.736593 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 2338
I20260619 16:49:14.742547 2781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:14.736808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39321 (local address 127.0.0.1:40706)
0619 16:49:14.742331 (+ 5523us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:14.742460 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5291,"thread_start_us":75,"threads_started":1}
W20260619 16:49:14.743047 2113 catalog_manager.cc:4584] TS bc51e846ab01481e823acbcb5b5a9211 (127.0.98.1:39321): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3597009a8b5d475bb8d31c7cf92b7f66: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39321: connect: Connection refused (error 111)
I20260619 16:49:14.746411 2781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:14.746258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41853 (local address 127.0.0.1:42620)
0619 16:49:14.746317 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:14.746351 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 16:49:14.746641 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 2478
I20260619 16:49:14.753311 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 2094
2026-06-19T16:49:14Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5914 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:49:14 dist-test-slave-jsw9 krb5kdc[2784](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:49:14 dist-test-slave-jsw9 krb5kdc[2784](info): set up 2 sockets
Jun 19 16:49:14 dist-test-slave-jsw9 krb5kdc[2784](info): commencing operation
krb5kdc: starting...
W20260619 16:49:15.770390 392 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:49:15 dist-test-slave-jsw9 krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:49:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:49:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:49:15.914683 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36923
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36923
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:16.014743 2797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:16.015064 2797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:16.015106 2797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:16.016810 2797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:16.016893 2797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:16.016956 2797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:16.016979 2797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:16.017005 2797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:49:16.019294 2797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:16.019703 2797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:16.019995 2797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:49:16.026126 2797 server_base.cc:1034] running on GCE node
W20260619 16:49:16.026031 2803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:16.026031 2805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:16.026038 2802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:16.026631 2797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:16.026996 2797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:16.028180 2797 hybrid_clock.cc:648] HybridClock initialized: now 1781887756028147 us; error 46 us; skew 500 ppm
Jun 19 16:49:16 dist-test-slave-jsw9 krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:16.031618 2797 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:49:16.032330 2797 webserver.cc:460] Webserver started at http://127.0.98.62:41991/ using document root <none> and password file <none>
I20260619 16:49:16.032612 2797 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:16.032680 2797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:16.032823 2797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:16.034118 2797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "4fd544b7179b4b8baa1a34fd43a33f69"
format_stamp: "Formatted at 2026-06-19 16:49:16 on dist-test-slave-jsw9"
server_key: "064e9564e61b3bab935102c050c055bc"
server_key_iv: "71998343f114a508b7d4f69fa6e1e2b2"
server_key_version: "encryptionkey@0"
I20260619 16:49:16.034571 2797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "4fd544b7179b4b8baa1a34fd43a33f69"
format_stamp: "Formatted at 2026-06-19 16:49:16 on dist-test-slave-jsw9"
server_key: "064e9564e61b3bab935102c050c055bc"
server_key_iv: "71998343f114a508b7d4f69fa6e1e2b2"
server_key_version: "encryptionkey@0"
I20260619 16:49:16.036177 2797 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 16:49:16.037111 2812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:16.037449 2797 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:16.037586 2797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "4fd544b7179b4b8baa1a34fd43a33f69"
format_stamp: "Formatted at 2026-06-19 16:49:16 on dist-test-slave-jsw9"
server_key: "064e9564e61b3bab935102c050c055bc"
server_key_iv: "71998343f114a508b7d4f69fa6e1e2b2"
server_key_version: "encryptionkey@0"
I20260619 16:49:16.037672 2797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:16.053790 2797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:16.059401 2797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:16.060727 2797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:16.067009 2797 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36923
I20260619 16:49:16.067047 2864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36923 every 8 connection(s)
I20260619 16:49:16.067670 2797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:16.068737 2865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:16.072237 2865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fd544b7179b4b8baa1a34fd43a33f69: Bootstrap starting.
I20260619 16:49:16.072283 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 2797
I20260619 16:49:16.072379 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:49:16.072587 392 external_mini_cluster.cc:1428] Setting key 2c64bf4ecc311181b97b28ea7aea7f96
I20260619 16:49:16.073088 2865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fd544b7179b4b8baa1a34fd43a33f69: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:16.073469 2865 log.cc:824] T 00000000000000000000000000000000 P 4fd544b7179b4b8baa1a34fd43a33f69: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:16.074368 2865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fd544b7179b4b8baa1a34fd43a33f69: No bootstrap required, opened a new log
I20260619 16:49:16.077037 2865 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fd544b7179b4b8baa1a34fd43a33f69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd544b7179b4b8baa1a34fd43a33f69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36923 } }
I20260619 16:49:16.077234 2865 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fd544b7179b4b8baa1a34fd43a33f69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:16.077258 2865 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fd544b7179b4b8baa1a34fd43a33f69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd544b7179b4b8baa1a34fd43a33f69, State: Initialized, Role: FOLLOWER
I20260619 16:49:16.077385 2865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fd544b7179b4b8baa1a34fd43a33f69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd544b7179b4b8baa1a34fd43a33f69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36923 } }
I20260619 16:49:16.077477 2865 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fd544b7179b4b8baa1a34fd43a33f69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:16.077523 2865 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fd544b7179b4b8baa1a34fd43a33f69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:16.077579 2865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fd544b7179b4b8baa1a34fd43a33f69 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:16.078471 2865 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fd544b7179b4b8baa1a34fd43a33f69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd544b7179b4b8baa1a34fd43a33f69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36923 } }
I20260619 16:49:16.078601 2865 leader_election.cc:302] T 00000000000000000000000000000000 P 4fd544b7179b4b8baa1a34fd43a33f69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fd544b7179b4b8baa1a34fd43a33f69; no voters:
I20260619 16:49:16.078814 2865 leader_election.cc:288] T 00000000000000000000000000000000 P 4fd544b7179b4b8baa1a34fd43a33f69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:49:16.078994 2870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fd544b7179b4b8baa1a34fd43a33f69 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:16.079231 2865 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fd544b7179b4b8baa1a34fd43a33f69 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:16.079407 2870 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fd544b7179b4b8baa1a34fd43a33f69 [term 1 LEADER]: Becoming Leader. State: Replica: 4fd544b7179b4b8baa1a34fd43a33f69, State: Running, Role: LEADER
I20260619 16:49:16.079602 2870 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fd544b7179b4b8baa1a34fd43a33f69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd544b7179b4b8baa1a34fd43a33f69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36923 } }
I20260619 16:49:16.080646 2871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fd544b7179b4b8baa1a34fd43a33f69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fd544b7179b4b8baa1a34fd43a33f69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd544b7179b4b8baa1a34fd43a33f69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36923 } } }
I20260619 16:49:16.080888 2871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fd544b7179b4b8baa1a34fd43a33f69 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:16.081641 2872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fd544b7179b4b8baa1a34fd43a33f69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fd544b7179b4b8baa1a34fd43a33f69. Latest consensus state: current_term: 1 leader_uuid: "4fd544b7179b4b8baa1a34fd43a33f69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd544b7179b4b8baa1a34fd43a33f69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36923 } } }
I20260619 16:49:16.081729 2872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fd544b7179b4b8baa1a34fd43a33f69 [sys.catalog]: This master's current role is: LEADER
W20260619 16:49:16.081343 2884 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4fd544b7179b4b8baa1a34fd43a33f69: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:49:16.081881 2884 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:49:16.081949 2885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 19 16:49:16 dist-test-slave-jsw9 krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:16.082254 2885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:16.085042 2885 catalog_manager.cc:1347] Generated new cluster ID: 88c1c545617e466680316a02d58e8a3d
I20260619 16:49:16.085176 2885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:49:16.087018 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:16.073197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49362 (local address 127.0.98.62:36923)
0619 16:49:16.073417 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:16.073429 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:16.073466 (+ 37us) server_negotiation.cc:407] Connection header received
0619 16:49:16.074093 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:16.074105 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:16.074214 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:16.074398 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:16.075145 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.076744 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.077948 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.078668 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.078803 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:16.082747 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:16.082753 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:16.082753 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:16.082770 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:16.084963 (+ 2193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:16.085587 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:16.085593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:16.085593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:16.085698 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:16.085897 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:16.085898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:16.085899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:16.086241 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:16.086422 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:16.086584 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 16:49:16.086757 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":94,"threads_started":1}
I20260619 16:49:16.108071 2885 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:49:16.108958 2885 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:16.118765 2885 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fd544b7179b4b8baa1a34fd43a33f69: Generated new TSK 0
I20260619 16:49:16.119119 2885 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:49:16.153944 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36923
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:16.267407 2891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:16.267728 2891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:16.267760 2891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:16.269625 2891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:16.269730 2891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:16.269770 2891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:49:16.272055 2891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:16.272398 2891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:16.272686 2891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:16.278704 2899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:16.278719 2897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:16.278704 2896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:16.278775 2891 server_base.cc:1034] running on GCE node
I20260619 16:49:16.279174 2891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:16.279466 2891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:16.280695 2891 hybrid_clock.cc:648] HybridClock initialized: now 1781887756280638 us; error 65 us; skew 500 ppm
Jun 19 16:49:16 dist-test-slave-jsw9 krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:16.284161 2891 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:49:16.284863 2891 webserver.cc:460] Webserver started at http://127.0.98.1:43379/ using document root <none> and password file <none>
I20260619 16:49:16.285131 2891 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:16.285192 2891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:16.285348 2891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:16.286744 2891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-0/data/instance:
uuid: "b4b50c2fbfe346eb8e1359fca1f091e2"
format_stamp: "Formatted at 2026-06-19 16:49:16 on dist-test-slave-jsw9"
server_key: "f76bd44fc7db3b074225fa5a1ddb0d79"
server_key_iv: "73a436e5adb65f44cdfddbf4ff42327b"
server_key_version: "encryptionkey@0"
I20260619 16:49:16.287151 2891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b4b50c2fbfe346eb8e1359fca1f091e2"
format_stamp: "Formatted at 2026-06-19 16:49:16 on dist-test-slave-jsw9"
server_key: "f76bd44fc7db3b074225fa5a1ddb0d79"
server_key_iv: "73a436e5adb65f44cdfddbf4ff42327b"
server_key_version: "encryptionkey@0"
I20260619 16:49:16.288758 2891 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 16:49:16.289765 2906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:16.290091 2891 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:16.290252 2891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-0/wal
uuid: "b4b50c2fbfe346eb8e1359fca1f091e2"
format_stamp: "Formatted at 2026-06-19 16:49:16 on dist-test-slave-jsw9"
server_key: "f76bd44fc7db3b074225fa5a1ddb0d79"
server_key_iv: "73a436e5adb65f44cdfddbf4ff42327b"
server_key_version: "encryptionkey@0"
I20260619 16:49:16.290354 2891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:16.311246 2891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:16.313810 2891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:16.313966 2891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:16.314258 2891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:16.314644 2891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:16.314682 2891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:16.314706 2891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:16.314718 2891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:16.321306 2891 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39399
I20260619 16:49:16.321376 3019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39399 every 8 connection(s)
I20260619 16:49:16.321728 2891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 16:49:16 dist-test-slave-jsw9 krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:16.331228 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 2891
I20260619 16:49:16.331336 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:49:16.331548 392 external_mini_cluster.cc:1428] Setting key dd41fe65edf1112d680fd07037f12753
I20260619 16:49:16.332072 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:16.322734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57151 (local address 127.0.98.62:36923)
0619 16:49:16.322823 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:16.322826 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:16.323032 (+ 206us) server_negotiation.cc:407] Connection header received
0619 16:49:16.324125 (+ 1093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:16.324127 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:16.324141 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:16.324196 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:16.324908 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.325463 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.326564 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.326792 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.326851 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:16.329245 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:16.329247 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:16.329247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:16.329256 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:16.330625 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:16.331041 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:16.331043 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:16.331043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:16.331099 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:16.331322 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:16.331323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:16.331324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:16.331470 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:16.331531 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:16.331856 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 16:49:16.331963 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 16:49:16.332279 3022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:16.322692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36923 (local address 127.0.98.1:57151)
0619 16:49:16.322866 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:16.322915 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:16.324007 (+ 1092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:16.324278 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:16.324283 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:16.324597 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:16.324818 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:16.324823 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.325549 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.325550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:16.326490 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:16.326492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.326863 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.326865 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:16.326995 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:16.327551 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:16.327560 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:16.329138 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:16.330706 (+ 1568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:16.330715 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:16.330718 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:16.330968 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:16.331151 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:16.331152 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:16.331152 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:16.331267 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:16.331527 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:16.331534 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:16.331727 (+ 193us) client_negotiation.cc:769] Sending connection context
0619 16:49:16.331831 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 16:49:16.332019 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":79,"threads_started":1}
I20260619 16:49:16.332882 3020 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36923
I20260619 16:49:16.332991 3020 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:16.333201 3020 heartbeater.cc:511] Master 127.0.98.62:36923 requested a full tablet report, sending...
I20260619 16:49:16.333770 2829 ts_manager.cc:194] Registered new tserver with Master: b4b50c2fbfe346eb8e1359fca1f091e2 (127.0.98.1:39399)
I20260619 16:49:16.334731 2829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57151
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:49:16.363504 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36923
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:16.456722 3025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:16.457017 3025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:16.457047 3025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:16.458564 3025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:16.458648 3025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:16.458685 3025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:49:16.460692 3025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:16.461047 3025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:16.461344 3025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:16.467551 3033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:16.467531 3030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:16.467556 3031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:16.467978 3025 server_base.cc:1034] running on GCE node
I20260619 16:49:16.468348 3025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:16.468722 3025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:16.469905 3025 hybrid_clock.cc:648] HybridClock initialized: now 1781887756469884 us; error 42 us; skew 500 ppm
Jun 19 16:49:16 dist-test-slave-jsw9 krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:16.473042 3025 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:49:16.473592 3025 webserver.cc:460] Webserver started at http://127.0.98.2:43791/ using document root <none> and password file <none>
I20260619 16:49:16.473810 3025 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:16.473858 3025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:16.473987 3025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:16.475248 3025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-1/data/instance:
uuid: "80b03424fd9c41ddbeb915200b651cd9"
format_stamp: "Formatted at 2026-06-19 16:49:16 on dist-test-slave-jsw9"
server_key: "5368ec87702c2edfc29be183b5be15fd"
server_key_iv: "a29e383308a0fe427854017f4ab13230"
server_key_version: "encryptionkey@0"
I20260619 16:49:16.475610 3025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-1/wal/instance:
uuid: "80b03424fd9c41ddbeb915200b651cd9"
format_stamp: "Formatted at 2026-06-19 16:49:16 on dist-test-slave-jsw9"
server_key: "5368ec87702c2edfc29be183b5be15fd"
server_key_iv: "a29e383308a0fe427854017f4ab13230"
server_key_version: "encryptionkey@0"
I20260619 16:49:16.477044 3025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 16:49:16.477864 3040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:16.478060 3025 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:16.478173 3025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-1/wal
uuid: "80b03424fd9c41ddbeb915200b651cd9"
format_stamp: "Formatted at 2026-06-19 16:49:16 on dist-test-slave-jsw9"
server_key: "5368ec87702c2edfc29be183b5be15fd"
server_key_iv: "a29e383308a0fe427854017f4ab13230"
server_key_version: "encryptionkey@0"
I20260619 16:49:16.478288 3025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:16.505601 3025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:16.508670 3025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:16.508874 3025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:16.509199 3025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:16.509848 3025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:16.509903 3025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:16.509941 3025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:16.509960 3025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:16.518092 3025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37667
I20260619 16:49:16.518216 3153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37667 every 8 connection(s)
I20260619 16:49:16.518604 3025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:49:16.520514 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 3025
I20260619 16:49:16.520622 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:49:16.520810 392 external_mini_cluster.cc:1428] Setting key 7942c6ad5a0604f5e8b1cba99f943fd7
Jun 19 16:49:16 dist-test-slave-jsw9 krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:16.530014 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:16.519511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56967 (local address 127.0.98.62:36923)
0619 16:49:16.519609 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:16.519610 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:16.519811 (+ 201us) server_negotiation.cc:407] Connection header received
0619 16:49:16.521269 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:16.521271 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:16.521287 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:16.521334 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:16.522254 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.522807 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.523973 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.524169 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.524230 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:16.527017 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:16.527019 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:16.527020 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:16.527029 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:16.528442 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:16.528952 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:16.528954 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:16.528955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:16.529013 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:16.529271 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:16.529272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:16.529273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:16.529408 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:16.529469 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:16.529799 (+ 330us) server_negotiation.cc:299] Negotiation successful
0619 16:49:16.529900 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:49:16.530357 3156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:16.519448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36923 (local address 127.0.98.2:56967)
0619 16:49:16.519637 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:16.519689 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:16.521071 (+ 1382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:16.521432 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:16.521438 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:16.521780 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:16.522126 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:16.522134 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.522904 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.522905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:16.523852 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:16.523855 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.524247 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.524248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:16.524395 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:16.525138 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:16.525149 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:16.526882 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:16.528542 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:16.528555 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:16.528559 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:16.528872 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:16.529069 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:16.529070 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:16.529070 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:16.529211 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:16.529466 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:16.529474 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:16.529662 (+ 188us) client_negotiation.cc:769] Sending connection context
0619 16:49:16.529770 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 16:49:16.529983 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 16:49:16.530885 3154 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36923
I20260619 16:49:16.531003 3154 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:16.531204 3154 heartbeater.cc:511] Master 127.0.98.62:36923 requested a full tablet report, sending...
I20260619 16:49:16.531628 2829 ts_manager.cc:194] Registered new tserver with Master: 80b03424fd9c41ddbeb915200b651cd9 (127.0.98.2:37667)
I20260619 16:49:16.532081 2829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56967
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:49:16.554957 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36923
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:16.651188 3159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:16.651473 3159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:16.651502 3159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:16.652889 3159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:16.652935 3159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:16.652966 3159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:49:16.655366 3159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:16.655679 3159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:16.655934 3159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:16.662521 3164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:16.662597 3159 server_base.cc:1034] running on GCE node
W20260619 16:49:16.662525 3165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:16.662521 3167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:16.663050 3159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:16.663465 3159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:16.664659 3159 hybrid_clock.cc:648] HybridClock initialized: now 1781887756664637 us; error 37 us; skew 500 ppm
Jun 19 16:49:16 dist-test-slave-jsw9 krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:16.667711 3159 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:49:16.668226 3159 webserver.cc:460] Webserver started at http://127.0.98.3:45213/ using document root <none> and password file <none>
I20260619 16:49:16.668463 3159 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:16.668526 3159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:16.668675 3159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:16.669893 3159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-2/data/instance:
uuid: "40348e4137064681a255379ba56c2598"
format_stamp: "Formatted at 2026-06-19 16:49:16 on dist-test-slave-jsw9"
server_key: "b72ae62051a988275ce9c7bf62a368d9"
server_key_iv: "d397c9aa32f9ad3aa971cd5bc4780011"
server_key_version: "encryptionkey@0"
I20260619 16:49:16.670315 3159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-2/wal/instance:
uuid: "40348e4137064681a255379ba56c2598"
format_stamp: "Formatted at 2026-06-19 16:49:16 on dist-test-slave-jsw9"
server_key: "b72ae62051a988275ce9c7bf62a368d9"
server_key_iv: "d397c9aa32f9ad3aa971cd5bc4780011"
server_key_version: "encryptionkey@0"
I20260619 16:49:16.671932 3159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:49:16.672806 3174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:16.672991 3159 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:16.673085 3159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-2/wal
uuid: "40348e4137064681a255379ba56c2598"
format_stamp: "Formatted at 2026-06-19 16:49:16 on dist-test-slave-jsw9"
server_key: "b72ae62051a988275ce9c7bf62a368d9"
server_key_iv: "d397c9aa32f9ad3aa971cd5bc4780011"
server_key_version: "encryptionkey@0"
I20260619 16:49:16.673156 3159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:16.690799 3159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:16.693224 3159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:16.693387 3159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:16.693650 3159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:16.693992 3159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:16.694031 3159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:16.694063 3159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:16.694087 3159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:16.701211 3159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43543
I20260619 16:49:16.701259 3287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43543 every 8 connection(s)
I20260619 16:49:16.701684 3159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:49:16.701972 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 3159
I20260619 16:49:16.702082 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:49:16.702356 392 external_mini_cluster.cc:1428] Setting key 9d00cc0a7b83a20d76c3ed95488942f3
Jun 19 16:49:16 dist-test-slave-jsw9 krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:16.713764 2868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:16.703090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40251 (local address 127.0.98.62:36923)
0619 16:49:16.703147 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:16.703149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:16.703689 (+ 540us) server_negotiation.cc:407] Connection header received
0619 16:49:16.705159 (+ 1470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:16.705161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:16.705176 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:16.705232 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:16.706052 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.706662 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.707795 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.708009 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.708072 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:16.710672 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:16.710675 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:16.710675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:16.710686 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:16.711980 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:16.712678 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:16.712680 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:16.712680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:16.712752 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:16.713016 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:16.713017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:16.713018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:16.713156 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:16.713222 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:16.713553 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 16:49:16.713655 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 16:49:16.713999 3290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:16.703189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36923 (local address 127.0.98.3:40251)
0619 16:49:16.703508 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:16.703593 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:16.705033 (+ 1440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:16.705346 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:16.705351 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:16.705687 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:16.705942 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:16.705948 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.706761 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.706762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:16.707698 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:16.707701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:16.708090 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:16.708091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:16.708243 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:16.708858 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:16.708869 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:16.710555 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:16.712074 (+ 1519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:16.712084 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:16.712087 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:16.712567 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:16.712816 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:16.712817 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:16.712818 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:16.712954 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:16.713219 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:16.713228 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:16.713414 (+ 186us) client_negotiation.cc:769] Sending connection context
0619 16:49:16.713521 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 16:49:16.713724 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":118,"threads_started":1}
I20260619 16:49:16.714901 3288 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36923
I20260619 16:49:16.715039 3288 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:16.715272 3288 heartbeater.cc:511] Master 127.0.98.62:36923 requested a full tablet report, sending...
I20260619 16:49:16.715729 2828 ts_manager.cc:194] Registered new tserver with Master: 40348e4137064681a255379ba56c2598 (127.0.98.3:43543)
I20260619 16:49:16.715999 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:49:16.716277 2828 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40251
W20260619 16:49:16.729285 2868 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:16.728167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49378 (local address 127.0.98.62:36923)
0619 16:49:16.728256 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:16.728258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:16.728364 (+ 106us) server_negotiation.cc:407] Connection header received
0619 16:49:16.728456 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:16.728457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:16.728469 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:16.728516 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:16.729189 (+ 673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49378: BlockingRecv error: recv got EOF from 127.0.0.1:49378 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 16:49:16.730223 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 2891
I20260619 16:49:16.734786 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 3025
I20260619 16:49:16.739432 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 3159
I20260619 16:49:16.743790 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 2797
2026-06-19T16:49:16Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1984 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:49:16 dist-test-slave-jsw9 krb5kdc[3300](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:49:16 dist-test-slave-jsw9 krb5kdc[3300](info): set up 2 sockets
Jun 19 16:49:16 dist-test-slave-jsw9 krb5kdc[3300](info): commencing operation
krb5kdc: starting...
W20260619 16:49:17.718644 392 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:49:17 dist-test-slave-jsw9 krb5kdc[3300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:49:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:49:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:49:17.866762 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42267
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:17.960430 3313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:17.960741 3313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:17.960780 3313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:17.962291 3313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:17.962350 3313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:17.962369 3313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:17.962387 3313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:17.962402 3313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:49:17.964298 3313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:17.964630 3313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:17.964890 3313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:17.970333 3321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:17.970391 3319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:17.970330 3318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:17.970551 3313 server_base.cc:1034] running on GCE node
I20260619 16:49:17.970922 3313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:17.971293 3313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:17.972456 3313 hybrid_clock.cc:648] HybridClock initialized: now 1781887757972424 us; error 56 us; skew 500 ppm
Jun 19 16:49:17 dist-test-slave-jsw9 krb5kdc[3300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:17.981693 3313 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:49:17.982617 3313 webserver.cc:460] Webserver started at http://127.0.98.62:35717/ using document root <none> and password file <none>
I20260619 16:49:17.982932 3313 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:17.982995 3313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:17.983148 3313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:17.984493 3313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "5e5ebbd26dce4deeaabc91c0ad3f02e7"
format_stamp: "Formatted at 2026-06-19 16:49:17 on dist-test-slave-jsw9"
server_key: "98a12282fb49d4e10cc890038f74761d"
server_key_iv: "36ce32c34a04341e014c1872b39390ed"
server_key_version: "encryptionkey@0"
I20260619 16:49:17.984892 3313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "5e5ebbd26dce4deeaabc91c0ad3f02e7"
format_stamp: "Formatted at 2026-06-19 16:49:17 on dist-test-slave-jsw9"
server_key: "98a12282fb49d4e10cc890038f74761d"
server_key_iv: "36ce32c34a04341e014c1872b39390ed"
server_key_version: "encryptionkey@0"
I20260619 16:49:17.986568 3313 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260619 16:49:17.987442 3328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:17.987672 3313 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:17.987776 3313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "5e5ebbd26dce4deeaabc91c0ad3f02e7"
format_stamp: "Formatted at 2026-06-19 16:49:17 on dist-test-slave-jsw9"
server_key: "98a12282fb49d4e10cc890038f74761d"
server_key_iv: "36ce32c34a04341e014c1872b39390ed"
server_key_version: "encryptionkey@0"
I20260619 16:49:17.987850 3313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:18.004700 3313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:18.007772 3313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:18.007990 3313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:18.012635 3313 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42267
I20260619 16:49:18.012669 3380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42267 every 8 connection(s)
I20260619 16:49:18.013085 3313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:18.013259 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 3313
I20260619 16:49:18.013350 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:49:18.013581 392 external_mini_cluster.cc:1428] Setting key b28b08a8d163fecb26e2ba29a55e5c37
I20260619 16:49:18.014214 3381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:18.017231 3381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e5ebbd26dce4deeaabc91c0ad3f02e7: Bootstrap starting.
I20260619 16:49:18.017946 3381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e5ebbd26dce4deeaabc91c0ad3f02e7: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:18.018301 3381 log.cc:824] T 00000000000000000000000000000000 P 5e5ebbd26dce4deeaabc91c0ad3f02e7: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:18.019100 3381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e5ebbd26dce4deeaabc91c0ad3f02e7: No bootstrap required, opened a new log
Jun 19 16:49:18 dist-test-slave-jsw9 krb5kdc[3300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:18.021395 3381 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e5ebbd26dce4deeaabc91c0ad3f02e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e5ebbd26dce4deeaabc91c0ad3f02e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42267 } }
I20260619 16:49:18.021587 3381 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e5ebbd26dce4deeaabc91c0ad3f02e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:18.021633 3381 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e5ebbd26dce4deeaabc91c0ad3f02e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e5ebbd26dce4deeaabc91c0ad3f02e7, State: Initialized, Role: FOLLOWER
I20260619 16:49:18.021790 3381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e5ebbd26dce4deeaabc91c0ad3f02e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e5ebbd26dce4deeaabc91c0ad3f02e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42267 } }
I20260619 16:49:18.021885 3381 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e5ebbd26dce4deeaabc91c0ad3f02e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:18.021931 3381 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e5ebbd26dce4deeaabc91c0ad3f02e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:18.021988 3381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e5ebbd26dce4deeaabc91c0ad3f02e7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:18.022840 3381 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e5ebbd26dce4deeaabc91c0ad3f02e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e5ebbd26dce4deeaabc91c0ad3f02e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42267 } }
I20260619 16:49:18.022950 3381 leader_election.cc:302] T 00000000000000000000000000000000 P 5e5ebbd26dce4deeaabc91c0ad3f02e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e5ebbd26dce4deeaabc91c0ad3f02e7; no voters:
I20260619 16:49:18.023175 3381 leader_election.cc:288] T 00000000000000000000000000000000 P 5e5ebbd26dce4deeaabc91c0ad3f02e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:49:18.023224 3386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e5ebbd26dce4deeaabc91c0ad3f02e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:18.023394 3386 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e5ebbd26dce4deeaabc91c0ad3f02e7 [term 1 LEADER]: Becoming Leader. State: Replica: 5e5ebbd26dce4deeaabc91c0ad3f02e7, State: Running, Role: LEADER
I20260619 16:49:18.023504 3381 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e5ebbd26dce4deeaabc91c0ad3f02e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:18.023562 3386 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e5ebbd26dce4deeaabc91c0ad3f02e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e5ebbd26dce4deeaabc91c0ad3f02e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42267 } }
I20260619 16:49:18.024205 3387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e5ebbd26dce4deeaabc91c0ad3f02e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e5ebbd26dce4deeaabc91c0ad3f02e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e5ebbd26dce4deeaabc91c0ad3f02e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42267 } } }
I20260619 16:49:18.024295 3387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e5ebbd26dce4deeaabc91c0ad3f02e7 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:18.024757 3395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:18.025161 3388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e5ebbd26dce4deeaabc91c0ad3f02e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e5ebbd26dce4deeaabc91c0ad3f02e7. Latest consensus state: current_term: 1 leader_uuid: "5e5ebbd26dce4deeaabc91c0ad3f02e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e5ebbd26dce4deeaabc91c0ad3f02e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42267 } } }
I20260619 16:49:18.025336 3388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e5ebbd26dce4deeaabc91c0ad3f02e7 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:18.025362 3395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:18.025898 3384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:18.014171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33118 (local address 127.0.98.62:42267)
0619 16:49:18.014405 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:18.014414 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:18.014445 (+ 31us) server_negotiation.cc:407] Connection header received
0619 16:49:18.014970 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:18.014979 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:18.015033 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:18.015175 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:18.015870 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:18.016867 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:18.017714 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:18.018261 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:18.018376 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:18.021258 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:18.021264 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:18.021265 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:18.021280 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:18.023506 (+ 2226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:18.023936 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:18.023940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:18.023941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:18.024063 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:18.024574 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:18.024576 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:18.024577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:18.024958 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:18.025130 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:18.025583 (+ 453us) server_negotiation.cc:299] Negotiation successful
0619 16:49:18.025722 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":116,"threads_started":1}
I20260619 16:49:18.027247 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:49:18.027498 3395 catalog_manager.cc:1347] Generated new cluster ID: 3c9947a9d9e545e38805eaeef7e57ab3
I20260619 16:49:18.027563 3395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:49:18.040403 3395 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:49:18.041282 3395 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:18.048964 3384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:18.045210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33120 (local address 127.0.98.62:42267)
0619 16:49:18.045266 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:18.045268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:18.045415 (+ 147us) server_negotiation.cc:407] Connection header received
0619 16:49:18.045483 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:18.045485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:18.045498 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:18.045545 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:18.045970 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:18.046558 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:18.047330 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:18.047551 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:18.047611 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:18.048307 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:18.048309 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:18.048309 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:18.048642 (+ 333us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:18.048665 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:18.048673 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:18.048742 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:18.048829 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 16:49:18.048865 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 16:49:18.053973 3395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e5ebbd26dce4deeaabc91c0ad3f02e7: Generated new TSK 0
I20260619 16:49:18.054342 3395 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:49:18.062405 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 3313
2026-06-19T16:49:18Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1316 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:49:18 dist-test-slave-jsw9 krb5kdc[3414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:49:18 dist-test-slave-jsw9 krb5kdc[3414](info): set up 2 sockets
Jun 19 16:49:18 dist-test-slave-jsw9 krb5kdc[3414](info): commencing operation
krb5kdc: starting...
W20260619 16:49:19.034456 392 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:49:19 dist-test-slave-jsw9 krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:49:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:49:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:49:19.175205 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37789
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37789
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:19.283370 3427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:19.283689 3427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:19.283718 3427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:19.285810 3427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:19.285919 3427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:19.285934 3427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:19.285948 3427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:19.285959 3427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:49:19.288250 3427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:19.288666 3427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:19.288959 3427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:19.296600 3433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:19.296891 3427 server_base.cc:1034] running on GCE node
W20260619 16:49:19.296650 3435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:19.296679 3432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:19.297605 3427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:19.298130 3427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:19.299433 3427 hybrid_clock.cc:648] HybridClock initialized: now 1781887759299411 us; error 42 us; skew 500 ppm
Jun 19 16:49:19 dist-test-slave-jsw9 krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:19.303367 3427 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:49:19.304078 3427 webserver.cc:460] Webserver started at http://127.0.98.62:42103/ using document root <none> and password file <none>
I20260619 16:49:19.304524 3427 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:19.305475 3427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:19.305708 3427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:19.307451 3427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "703b8e979e4d4a4aaedb2a9799507773"
format_stamp: "Formatted at 2026-06-19 16:49:19 on dist-test-slave-jsw9"
server_key: "0f3cad193f45c877a7e3015987024460"
server_key_iv: "d1199e08be2541ee7294b0ac4c4cc112"
server_key_version: "encryptionkey@0"
I20260619 16:49:19.307931 3427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "703b8e979e4d4a4aaedb2a9799507773"
format_stamp: "Formatted at 2026-06-19 16:49:19 on dist-test-slave-jsw9"
server_key: "0f3cad193f45c877a7e3015987024460"
server_key_iv: "d1199e08be2541ee7294b0ac4c4cc112"
server_key_version: "encryptionkey@0"
I20260619 16:49:19.310469 3427 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260619 16:49:19.311724 3442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:19.312160 3427 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:19.312809 3427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "703b8e979e4d4a4aaedb2a9799507773"
format_stamp: "Formatted at 2026-06-19 16:49:19 on dist-test-slave-jsw9"
server_key: "0f3cad193f45c877a7e3015987024460"
server_key_iv: "d1199e08be2541ee7294b0ac4c4cc112"
server_key_version: "encryptionkey@0"
I20260619 16:49:19.312940 3427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:19.336014 3427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:19.338769 3427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:19.338918 3427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:19.343963 3427 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37789
I20260619 16:49:19.344225 3494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37789 every 8 connection(s)
I20260619 16:49:19.344470 3427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:19.345738 3495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:19.349097 3495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 703b8e979e4d4a4aaedb2a9799507773: Bootstrap starting.
I20260619 16:49:19.349866 3495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 703b8e979e4d4a4aaedb2a9799507773: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:19.350170 3495 log.cc:824] T 00000000000000000000000000000000 P 703b8e979e4d4a4aaedb2a9799507773: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:19.351217 3495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 703b8e979e4d4a4aaedb2a9799507773: No bootstrap required, opened a new log
I20260619 16:49:19.353061 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 3427
I20260619 16:49:19.353199 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:49:19.353477 392 external_mini_cluster.cc:1428] Setting key 25168733156fe25d8dc92b73ad286e4a
I20260619 16:49:19.353726 3495 raft_consensus.cc:348] T 00000000000000000000000000000000 P 703b8e979e4d4a4aaedb2a9799507773 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "703b8e979e4d4a4aaedb2a9799507773" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37789 } }
I20260619 16:49:19.353945 3495 raft_consensus.cc:374] T 00000000000000000000000000000000 P 703b8e979e4d4a4aaedb2a9799507773 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:19.353981 3495 raft_consensus.cc:729] T 00000000000000000000000000000000 P 703b8e979e4d4a4aaedb2a9799507773 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 703b8e979e4d4a4aaedb2a9799507773, State: Initialized, Role: FOLLOWER
I20260619 16:49:19.354211 3495 consensus_queue.cc:260] T 00000000000000000000000000000000 P 703b8e979e4d4a4aaedb2a9799507773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "703b8e979e4d4a4aaedb2a9799507773" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37789 } }
I20260619 16:49:19.354383 3495 raft_consensus.cc:388] T 00000000000000000000000000000000 P 703b8e979e4d4a4aaedb2a9799507773 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:19.354435 3495 raft_consensus.cc:482] T 00000000000000000000000000000000 P 703b8e979e4d4a4aaedb2a9799507773 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:19.354488 3495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 703b8e979e4d4a4aaedb2a9799507773 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:19.355607 3495 raft_consensus.cc:504] T 00000000000000000000000000000000 P 703b8e979e4d4a4aaedb2a9799507773 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "703b8e979e4d4a4aaedb2a9799507773" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37789 } }
I20260619 16:49:19.355732 3495 leader_election.cc:302] T 00000000000000000000000000000000 P 703b8e979e4d4a4aaedb2a9799507773 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 703b8e979e4d4a4aaedb2a9799507773; no voters:
I20260619 16:49:19.355926 3495 leader_election.cc:288] T 00000000000000000000000000000000 P 703b8e979e4d4a4aaedb2a9799507773 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:49:19.355978 3500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 703b8e979e4d4a4aaedb2a9799507773 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:19.356112 3500 raft_consensus.cc:686] T 00000000000000000000000000000000 P 703b8e979e4d4a4aaedb2a9799507773 [term 1 LEADER]: Becoming Leader. State: Replica: 703b8e979e4d4a4aaedb2a9799507773, State: Running, Role: LEADER
I20260619 16:49:19.356261 3495 sys_catalog.cc:564] T 00000000000000000000000000000000 P 703b8e979e4d4a4aaedb2a9799507773 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:19.356267 3500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 703b8e979e4d4a4aaedb2a9799507773 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "703b8e979e4d4a4aaedb2a9799507773" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37789 } }
I20260619 16:49:19.356860 3502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 703b8e979e4d4a4aaedb2a9799507773 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "703b8e979e4d4a4aaedb2a9799507773" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "703b8e979e4d4a4aaedb2a9799507773" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37789 } } }
I20260619 16:49:19.357038 3502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 703b8e979e4d4a4aaedb2a9799507773 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:19.357323 3501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 703b8e979e4d4a4aaedb2a9799507773 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 703b8e979e4d4a4aaedb2a9799507773. Latest consensus state: current_term: 1 leader_uuid: "703b8e979e4d4a4aaedb2a9799507773" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "703b8e979e4d4a4aaedb2a9799507773" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37789 } } }
I20260619 16:49:19.357445 3501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 703b8e979e4d4a4aaedb2a9799507773 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:19.357642 3509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:19.358390 3509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:19.360805 3509 catalog_manager.cc:1347] Generated new cluster ID: a9728cc02def45499d0f565e879d56f4
I20260619 16:49:19.360947 3509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 16:49:19 dist-test-slave-jsw9 krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:19.366654 3499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:19.354146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56708 (local address 127.0.98.62:37789)
0619 16:49:19.354402 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:19.354415 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:19.354452 (+ 37us) server_negotiation.cc:407] Connection header received
0619 16:49:19.354604 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:19.354612 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:19.354669 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:19.354928 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:19.355636 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:19.357007 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:19.358213 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:19.358863 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:19.359012 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:19.361990 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:19.361997 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:19.361997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:19.362014 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:19.364368 (+ 2354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:19.364863 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:19.364868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:19.364869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:19.365059 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:19.365279 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:19.365280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:19.365281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:19.365784 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:19.366056 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:19.366152 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 16:49:19.366378 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":139,"threads_started":1}
I20260619 16:49:19.373808 3509 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:49:19.374557 3509 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:19.380641 3509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 703b8e979e4d4a4aaedb2a9799507773: Generated new TSK 0
I20260619 16:49:19.381004 3509 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:49:19.413602 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37789
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:19.513167 3521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:19.513450 3521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:19.513481 3521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:19.515007 3521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:19.515067 3521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:19.515100 3521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:49:19.516932 3521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:19.517261 3521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:19.517549 3521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:19.523564 3529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:19.523590 3526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:19.523564 3527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:19.523737 3521 server_base.cc:1034] running on GCE node
I20260619 16:49:19.524093 3521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:19.524411 3521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:19.525594 3521 hybrid_clock.cc:648] HybridClock initialized: now 1781887759525607 us; error 75 us; skew 500 ppm
Jun 19 16:49:19 dist-test-slave-jsw9 krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:19.528860 3521 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:49:19.529822 3521 webserver.cc:460] Webserver started at http://127.0.98.1:37519/ using document root <none> and password file <none>
I20260619 16:49:19.530082 3521 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:19.530130 3521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:19.530299 3521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:19.531762 3521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-0/data/instance:
uuid: "5ea4b1c0add046fea379db24af2885ad"
format_stamp: "Formatted at 2026-06-19 16:49:19 on dist-test-slave-jsw9"
server_key: "1fd82092f7c794a9a849fd22c2741c2c"
server_key_iv: "6a14a12974f91d03aba98f73d9fca57b"
server_key_version: "encryptionkey@0"
I20260619 16:49:19.532183 3521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5ea4b1c0add046fea379db24af2885ad"
format_stamp: "Formatted at 2026-06-19 16:49:19 on dist-test-slave-jsw9"
server_key: "1fd82092f7c794a9a849fd22c2741c2c"
server_key_iv: "6a14a12974f91d03aba98f73d9fca57b"
server_key_version: "encryptionkey@0"
I20260619 16:49:19.533829 3521 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 16:49:19.534935 3536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:19.535202 3521 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 16:49:19.535303 3521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-0/wal
uuid: "5ea4b1c0add046fea379db24af2885ad"
format_stamp: "Formatted at 2026-06-19 16:49:19 on dist-test-slave-jsw9"
server_key: "1fd82092f7c794a9a849fd22c2741c2c"
server_key_iv: "6a14a12974f91d03aba98f73d9fca57b"
server_key_version: "encryptionkey@0"
I20260619 16:49:19.535385 3521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:19.554656 3521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:19.558151 3521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:19.558377 3521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:19.558656 3521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:19.559029 3521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:19.559067 3521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:19.559091 3521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:19.559105 3521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:19.566552 3521 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36175
I20260619 16:49:19.566587 3649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36175 every 8 connection(s)
I20260619 16:49:19.567065 3521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:49:19.570331 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 3521
I20260619 16:49:19.570449 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:49:19.570668 392 external_mini_cluster.cc:1428] Setting key 35f20ab8ddedbe838263d708e85e3606
Jun 19 16:49:19 dist-test-slave-jsw9 krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:19.580312 3499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:19.567971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45713 (local address 127.0.98.62:37789)
0619 16:49:19.568039 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:19.568041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:19.568319 (+ 278us) server_negotiation.cc:407] Connection header received
0619 16:49:19.568914 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:19.568915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:19.568927 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:19.568996 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:19.570156 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:19.571021 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:19.572738 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:19.573046 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:19.573140 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:19.576318 (+ 3178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:19.576320 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:19.576320 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:19.576333 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:19.578083 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:19.578708 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:19.578709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:19.578710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:19.578785 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:19.579275 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:19.579276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:19.579277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:19.579532 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:19.579618 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:19.580077 (+ 459us) server_negotiation.cc:299] Negotiation successful
0619 16:49:19.580195 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 16:49:19.580570 3652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:19.567938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37789 (local address 127.0.98.1:45713)
0619 16:49:19.568149 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:19.568200 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:19.568794 (+ 594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:19.569074 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:19.569078 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:19.569750 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:19.570011 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:19.570017 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:19.571155 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:19.571156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:19.572602 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:19.572607 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:19.573276 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:19.573279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:19.573508 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:19.574175 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:19.574375 (+ 200us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:19.576171 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:19.578256 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:19.578271 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:19.578276 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:19.578587 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:19.578904 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:19.578905 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:19.578906 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:19.579174 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:19.579618 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:19.579627 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:19.579864 (+ 237us) client_negotiation.cc:769] Sending connection context
0619 16:49:19.580028 (+ 164us) client_negotiation.cc:240] Negotiation successful
0619 16:49:19.580301 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":78,"threads_started":1}
I20260619 16:49:19.581259 3650 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37789
I20260619 16:49:19.581383 3650 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:19.581609 3650 heartbeater.cc:511] Master 127.0.98.62:37789 requested a full tablet report, sending...
I20260619 16:49:19.582369 3459 ts_manager.cc:194] Registered new tserver with Master: 5ea4b1c0add046fea379db24af2885ad (127.0.98.1:36175)
I20260619 16:49:19.583225 3459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45713
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:49:19.605355 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37789
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:19.706735 3655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:19.707031 3655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:19.707062 3655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:19.708667 3655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:19.708740 3655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:19.708774 3655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:49:19.710695 3655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:19.711016 3655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:19.711288 3655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:49:19.717892 3655 server_base.cc:1034] running on GCE node
W20260619 16:49:19.717832 3663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:19.717831 3660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:19.717832 3661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:19.718353 3655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:19.718626 3655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:19.719780 3655 hybrid_clock.cc:648] HybridClock initialized: now 1781887759719767 us; error 33 us; skew 500 ppm
Jun 19 16:49:19 dist-test-slave-jsw9 krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:19.722747 3655 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:49:19.723443 3655 webserver.cc:460] Webserver started at http://127.0.98.2:39401/ using document root <none> and password file <none>
I20260619 16:49:19.723707 3655 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:19.723762 3655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:19.723902 3655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:19.725157 3655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-1/data/instance:
uuid: "7dc9ea82384c489f85ca3f008b0c0928"
format_stamp: "Formatted at 2026-06-19 16:49:19 on dist-test-slave-jsw9"
server_key: "9f58a15a819bdeb1d61a9423eed8c307"
server_key_iv: "ec98facbdda749e34127a6b5b33ef668"
server_key_version: "encryptionkey@0"
I20260619 16:49:19.725529 3655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7dc9ea82384c489f85ca3f008b0c0928"
format_stamp: "Formatted at 2026-06-19 16:49:19 on dist-test-slave-jsw9"
server_key: "9f58a15a819bdeb1d61a9423eed8c307"
server_key_iv: "ec98facbdda749e34127a6b5b33ef668"
server_key_version: "encryptionkey@0"
I20260619 16:49:19.727051 3655 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 16:49:19.727926 3670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:19.728160 3655 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:49:19.728253 3655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-1/wal
uuid: "7dc9ea82384c489f85ca3f008b0c0928"
format_stamp: "Formatted at 2026-06-19 16:49:19 on dist-test-slave-jsw9"
server_key: "9f58a15a819bdeb1d61a9423eed8c307"
server_key_iv: "ec98facbdda749e34127a6b5b33ef668"
server_key_version: "encryptionkey@0"
I20260619 16:49:19.728328 3655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:19.752456 3655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:19.756208 3655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:19.756501 3655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:19.756829 3655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:19.757280 3655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:19.757323 3655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:19.757356 3655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:19.757380 3655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:19.764465 3655 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35387
I20260619 16:49:19.764532 3783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35387 every 8 connection(s)
I20260619 16:49:19.764897 3655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 16:49:19 dist-test-slave-jsw9 krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:19.772783 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 3655
I20260619 16:49:19.772921 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:49:19.773164 392 external_mini_cluster.cc:1428] Setting key b5728b70abb1f49bfc30be09c4f2e92d
I20260619 16:49:19.776813 3499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:19.765743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38407 (local address 127.0.98.62:37789)
0619 16:49:19.765840 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:19.765847 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:19.766025 (+ 178us) server_negotiation.cc:407] Connection header received
0619 16:49:19.766699 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:19.766701 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:19.766715 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:19.766773 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:19.767919 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:19.768468 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:19.769606 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:19.769814 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:19.769876 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:19.772677 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:19.772679 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:19.772680 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:19.772690 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:19.774984 (+ 2294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:19.775472 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:19.775473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:19.775474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:19.775560 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:19.775876 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:19.775877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:19.775877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:19.776058 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:19.776127 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:19.776622 (+ 495us) server_negotiation.cc:299] Negotiation successful
0619 16:49:19.776729 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:49:19.777187 3786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:19.765697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37789 (local address 127.0.98.2:38407)
0619 16:49:19.765877 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:19.765925 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:19.766570 (+ 645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:19.766870 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:19.766875 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:19.767550 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:19.767799 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:19.767805 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:19.768576 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:19.768577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:19.769502 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:19.769505 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:19.769892 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:19.769893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:19.770042 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:19.770688 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:19.770698 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:19.772541 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:19.775094 (+ 2553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:19.775103 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:19.775106 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:19.775396 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:19.775649 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:19.775650 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:19.775650 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:19.775801 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:19.776124 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:19.776132 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:19.776355 (+ 223us) client_negotiation.cc:769] Sending connection context
0619 16:49:19.776605 (+ 250us) client_negotiation.cc:240] Negotiation successful
0619 16:49:19.776887 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":76,"threads_started":1}
I20260619 16:49:19.777765 3784 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37789
I20260619 16:49:19.777875 3784 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:19.778084 3784 heartbeater.cc:511] Master 127.0.98.62:37789 requested a full tablet report, sending...
I20260619 16:49:19.778775 3459 ts_manager.cc:194] Registered new tserver with Master: 7dc9ea82384c489f85ca3f008b0c0928 (127.0.98.2:35387)
I20260619 16:49:19.779453 3459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38407
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:49:19.810604 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37789
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:19.912261 3789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:19.912556 3789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:19.912598 3789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:19.914637 3789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:19.914710 3789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:19.914760 3789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:49:19.916916 3789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:19.917289 3789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:19.917572 3789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:19.923640 3794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:19.923815 3789 server_base.cc:1034] running on GCE node
W20260619 16:49:19.923714 3797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:19.923625 3795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:19.924218 3789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:19.924525 3789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:19.925702 3789 hybrid_clock.cc:648] HybridClock initialized: now 1781887759925686 us; error 29 us; skew 500 ppm
Jun 19 16:49:19 dist-test-slave-jsw9 krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:19.928944 3789 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:49:19.929494 3789 webserver.cc:460] Webserver started at http://127.0.98.3:43247/ using document root <none> and password file <none>
I20260619 16:49:19.929733 3789 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:19.929788 3789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:19.929929 3789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:19.931327 3789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-2/data/instance:
uuid: "3961cf8a58b74cdfbbf5e40783bebfa1"
format_stamp: "Formatted at 2026-06-19 16:49:19 on dist-test-slave-jsw9"
server_key: "f5d28904e7884c1b0a62ac33ac7eb1b9"
server_key_iv: "202f96c6a592da7fbf8d782a2521232f"
server_key_version: "encryptionkey@0"
I20260619 16:49:19.931772 3789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3961cf8a58b74cdfbbf5e40783bebfa1"
format_stamp: "Formatted at 2026-06-19 16:49:19 on dist-test-slave-jsw9"
server_key: "f5d28904e7884c1b0a62ac33ac7eb1b9"
server_key_iv: "202f96c6a592da7fbf8d782a2521232f"
server_key_version: "encryptionkey@0"
I20260619 16:49:19.933413 3789 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 16:49:19.934291 3804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:19.934507 3789 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:49:19.934607 3789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-2/wal
uuid: "3961cf8a58b74cdfbbf5e40783bebfa1"
format_stamp: "Formatted at 2026-06-19 16:49:19 on dist-test-slave-jsw9"
server_key: "f5d28904e7884c1b0a62ac33ac7eb1b9"
server_key_iv: "202f96c6a592da7fbf8d782a2521232f"
server_key_version: "encryptionkey@0"
I20260619 16:49:19.934717 3789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:19.948437 3789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:19.951102 3789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:19.951273 3789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:19.951537 3789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:19.951923 3789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:19.951965 3789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:19.951999 3789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:19.952023 3789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:19.959846 3789 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34953
I20260619 16:49:19.959888 3917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34953 every 8 connection(s)
I20260619 16:49:19.960268 3789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 16:49:19 dist-test-slave-jsw9 krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:19.967890 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 3789
I20260619 16:49:19.968019 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781887741903642-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:49:19.968223 392 external_mini_cluster.cc:1428] Setting key dff8a32ecda266312048861986549b93
I20260619 16:49:19.972010 3499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:19.961152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48769 (local address 127.0.98.62:37789)
0619 16:49:19.961240 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:19.961243 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:19.961455 (+ 212us) server_negotiation.cc:407] Connection header received
0619 16:49:19.962159 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:19.962161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:19.962173 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:19.962265 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:19.963409 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:19.963980 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:19.965246 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:19.965476 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:19.965545 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:19.968245 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:19.968247 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:19.968248 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:19.968257 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:19.970141 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:19.970791 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:19.970793 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:19.970793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:19.970882 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:19.971185 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:19.971186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:19.971186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:19.971350 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:19.971420 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:19.971832 (+ 412us) server_negotiation.cc:299] Negotiation successful
0619 16:49:19.971928 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 16:49:19.972187 3920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:19.961099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37789 (local address 127.0.98.3:48769)
0619 16:49:19.961312 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:19.961360 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:19.962028 (+ 668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:19.962338 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:19.962342 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:19.963042 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:19.963299 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:19.963305 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:19.964085 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:19.964086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:19.965135 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:19.965139 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:19.965585 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:19.965586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:19.965727 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:19.966398 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:19.966409 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:19.968092 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:19.970378 (+ 2286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:19.970387 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:19.970391 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:19.970694 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:19.970943 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:19.970944 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:19.970944 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:19.971063 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:19.971437 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:19.971444 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:19.971661 (+ 217us) client_negotiation.cc:769] Sending connection context
0619 16:49:19.971778 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 16:49:19.971946 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":99,"threads_started":1}
I20260619 16:49:19.972952 3918 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37789
I20260619 16:49:19.973090 3918 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:19.973398 3918 heartbeater.cc:511] Master 127.0.98.62:37789 requested a full tablet report, sending...
I20260619 16:49:19.973901 3459 ts_manager.cc:194] Registered new tserver with Master: 3961cf8a58b74cdfbbf5e40783bebfa1 (127.0.98.3:34953)
I20260619 16:49:19.974390 3459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48769
I20260619 16:49:19.981215 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 16:49:19 dist-test-slave-jsw9 krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887759, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 16:49:20 dist-test-slave-jsw9 krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887759, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:20.015409 3499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:20.002359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56710 (local address 127.0.98.62:37789)
0619 16:49:20.002461 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:20.002464 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:20.002549 (+ 85us) server_negotiation.cc:407] Connection header received
0619 16:49:20.002692 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:20.002694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:20.002707 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:20.002759 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:20.003453 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:20.003994 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:20.005149 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:20.005527 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:20.005684 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:20.010872 (+ 5188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:20.010874 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:20.010875 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:20.010888 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:20.013381 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:20.014290 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:20.014291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:20.014292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:20.014378 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:20.014587 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:20.014589 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:20.014589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:20.014894 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:20.015025 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:20.015176 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 16:49:20.015296 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 16:49:20.015686 3459 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56710
I20260619 16:49:20.016363 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 3521
I20260619 16:49:20.023401 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 3655
I20260619 16:49:20.029865 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 3789
I20260619 16:49:20.035329 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 3427
2026-06-19T16:49:20Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1977 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:49:20 dist-test-slave-jsw9 krb5kdc[3930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:49:20 dist-test-slave-jsw9 krb5kdc[3930](info): set up 2 sockets
Jun 19 16:49:20 dist-test-slave-jsw9 krb5kdc[3930](info): commencing operation
krb5kdc: starting...
W20260619 16:49:21.054361 392 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:49:21 dist-test-slave-jsw9 krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:49:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:49:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:49:21.190308 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43055
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43055
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:21.292325 3943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:21.292654 3943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:21.292696 3943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:21.294148 3943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:21.294250 3943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:21.294274 3943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:21.294293 3943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:21.294312 3943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:49:21.296140 3943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:21.296463 3943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:21.296736 3943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:21.302614 3948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:21.302634 3949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:21.302611 3951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:21.303154 3943 server_base.cc:1034] running on GCE node
I20260619 16:49:21.303514 3943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:21.303911 3943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:21.305631 3943 hybrid_clock.cc:648] HybridClock initialized: now 1781887761305593 us; error 54 us; skew 500 ppm
Jun 19 16:49:21 dist-test-slave-jsw9 krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:21.309928 3943 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:49:21.310832 3943 webserver.cc:460] Webserver started at http://127.0.98.62:33819/ using document root <none> and password file <none>
I20260619 16:49:21.311111 3943 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:21.311173 3943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:21.311432 3943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:21.312948 3943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "c1be3fea39d34e769fe9c939d2b44bf5"
format_stamp: "Formatted at 2026-06-19 16:49:21 on dist-test-slave-jsw9"
server_key: "56eafd1b30a3af767fd8dff56b7e8b56"
server_key_iv: "5a5e0892bc36738db4d11d831ad6a7bd"
server_key_version: "encryptionkey@0"
I20260619 16:49:21.313588 3943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "c1be3fea39d34e769fe9c939d2b44bf5"
format_stamp: "Formatted at 2026-06-19 16:49:21 on dist-test-slave-jsw9"
server_key: "56eafd1b30a3af767fd8dff56b7e8b56"
server_key_iv: "5a5e0892bc36738db4d11d831ad6a7bd"
server_key_version: "encryptionkey@0"
I20260619 16:49:21.315445 3943 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260619 16:49:21.316538 3958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:21.316946 3943 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:49:21.317068 3943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "c1be3fea39d34e769fe9c939d2b44bf5"
format_stamp: "Formatted at 2026-06-19 16:49:21 on dist-test-slave-jsw9"
server_key: "56eafd1b30a3af767fd8dff56b7e8b56"
server_key_iv: "5a5e0892bc36738db4d11d831ad6a7bd"
server_key_version: "encryptionkey@0"
I20260619 16:49:21.317153 3943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:21.342576 3943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:21.346243 3943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:21.346437 3943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:21.352115 3943 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43055
I20260619 16:49:21.352273 4010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43055 every 8 connection(s)
I20260619 16:49:21.352630 3943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:21.353865 4011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:21.356891 4011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1be3fea39d34e769fe9c939d2b44bf5: Bootstrap starting.
I20260619 16:49:21.357108 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 3943
I20260619 16:49:21.357239 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:49:21.357424 392 external_mini_cluster.cc:1428] Setting key 7cc0d7311a89855c55f2f5df4154a17c
I20260619 16:49:21.357720 4011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1be3fea39d34e769fe9c939d2b44bf5: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:21.358109 4011 log.cc:824] T 00000000000000000000000000000000 P c1be3fea39d34e769fe9c939d2b44bf5: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:21.359054 4011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1be3fea39d34e769fe9c939d2b44bf5: No bootstrap required, opened a new log
I20260619 16:49:21.361157 4011 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1be3fea39d34e769fe9c939d2b44bf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1be3fea39d34e769fe9c939d2b44bf5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43055 } }
I20260619 16:49:21.361327 4011 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1be3fea39d34e769fe9c939d2b44bf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:21.361361 4011 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1be3fea39d34e769fe9c939d2b44bf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1be3fea39d34e769fe9c939d2b44bf5, State: Initialized, Role: FOLLOWER
I20260619 16:49:21.361508 4011 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1be3fea39d34e769fe9c939d2b44bf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1be3fea39d34e769fe9c939d2b44bf5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43055 } }
I20260619 16:49:21.361590 4011 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1be3fea39d34e769fe9c939d2b44bf5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:21.361625 4011 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1be3fea39d34e769fe9c939d2b44bf5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:21.361678 4011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1be3fea39d34e769fe9c939d2b44bf5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:21.362437 4011 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1be3fea39d34e769fe9c939d2b44bf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1be3fea39d34e769fe9c939d2b44bf5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43055 } }
I20260619 16:49:21.362551 4011 leader_election.cc:302] T 00000000000000000000000000000000 P c1be3fea39d34e769fe9c939d2b44bf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1be3fea39d34e769fe9c939d2b44bf5; no voters:
I20260619 16:49:21.362733 4011 leader_election.cc:288] T 00000000000000000000000000000000 P c1be3fea39d34e769fe9c939d2b44bf5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:49:21.362918 4017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1be3fea39d34e769fe9c939d2b44bf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:21.363055 4011 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1be3fea39d34e769fe9c939d2b44bf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:21.363122 4017 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1be3fea39d34e769fe9c939d2b44bf5 [term 1 LEADER]: Becoming Leader. State: Replica: c1be3fea39d34e769fe9c939d2b44bf5, State: Running, Role: LEADER
I20260619 16:49:21.363232 4017 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1be3fea39d34e769fe9c939d2b44bf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1be3fea39d34e769fe9c939d2b44bf5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43055 } }
I20260619 16:49:21.363696 4018 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1be3fea39d34e769fe9c939d2b44bf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1be3fea39d34e769fe9c939d2b44bf5. Latest consensus state: current_term: 1 leader_uuid: "c1be3fea39d34e769fe9c939d2b44bf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1be3fea39d34e769fe9c939d2b44bf5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43055 } } }
I20260619 16:49:21.363802 4018 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1be3fea39d34e769fe9c939d2b44bf5 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:21.364156 4025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 19 16:49:21 dist-test-slave-jsw9 krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:21.365420 4016 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1be3fea39d34e769fe9c939d2b44bf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1be3fea39d34e769fe9c939d2b44bf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1be3fea39d34e769fe9c939d2b44bf5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43055 } } }
I20260619 16:49:21.365525 4025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:21.365614 4016 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1be3fea39d34e769fe9c939d2b44bf5 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:21.368616 4025 catalog_manager.cc:1347] Generated new cluster ID: 34db84a0bde4456dba167accf1dd5757
I20260619 16:49:21.368687 4025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:49:21.372171 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:21.358090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49666 (local address 127.0.98.62:43055)
0619 16:49:21.358330 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:21.358341 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:21.358375 (+ 34us) server_negotiation.cc:407] Connection header received
0619 16:49:21.358990 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:21.359000 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:21.359070 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:21.359259 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:21.359933 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:21.360896 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:21.361810 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:21.362338 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:21.362445 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:21.366554 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:21.366560 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:21.366561 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:21.366578 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:21.369745 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:21.370375 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:21.370379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:21.370381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:21.370518 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:21.370781 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:21.370782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:21.370782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:21.371175 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:21.371369 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:21.371674 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 16:49:21.371875 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":101,"threads_started":1}
I20260619 16:49:21.376315 4025 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:49:21.377331 4025 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:21.381470 4025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1be3fea39d34e769fe9c939d2b44bf5: Generated new TSK 0
I20260619 16:49:21.381721 4025 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:49:21.419796 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43055
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:21.568250 4037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:21.568609 4037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:21.568651 4037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:21.571399 4037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:21.571525 4037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:21.571578 4037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:49:21.574347 4037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:21.574682 4037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:21.574935 4037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:21.581476 4042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:21.581638 4037 server_base.cc:1034] running on GCE node
W20260619 16:49:21.581518 4045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:21.581846 4043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:21.582134 4037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:21.582486 4037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:21.583691 4037 hybrid_clock.cc:648] HybridClock initialized: now 1781887761583674 us; error 34 us; skew 500 ppm
Jun 19 16:49:21 dist-test-slave-jsw9 krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:21.587627 4037 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:49:21.588361 4037 webserver.cc:460] Webserver started at http://127.0.98.1:44453/ using document root <none> and password file <none>
I20260619 16:49:21.588644 4037 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:21.588707 4037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:21.588866 4037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:21.590180 4037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-0/data/instance:
uuid: "a00c27b4d19f4d829be84a4cc2a07f58"
format_stamp: "Formatted at 2026-06-19 16:49:21 on dist-test-slave-jsw9"
server_key: "9461e0f3cfe0c6422c17ec6d4f453bf6"
server_key_iv: "3a0501e4a011b667323a7847412527db"
server_key_version: "encryptionkey@0"
I20260619 16:49:21.590649 4037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a00c27b4d19f4d829be84a4cc2a07f58"
format_stamp: "Formatted at 2026-06-19 16:49:21 on dist-test-slave-jsw9"
server_key: "9461e0f3cfe0c6422c17ec6d4f453bf6"
server_key_iv: "3a0501e4a011b667323a7847412527db"
server_key_version: "encryptionkey@0"
I20260619 16:49:21.592187 4037 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:21.593055 4052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:21.593300 4037 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:21.593403 4037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-0/wal
uuid: "a00c27b4d19f4d829be84a4cc2a07f58"
format_stamp: "Formatted at 2026-06-19 16:49:21 on dist-test-slave-jsw9"
server_key: "9461e0f3cfe0c6422c17ec6d4f453bf6"
server_key_iv: "3a0501e4a011b667323a7847412527db"
server_key_version: "encryptionkey@0"
I20260619 16:49:21.593479 4037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:21.611197 4037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:21.613732 4037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:21.613924 4037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:21.614290 4037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:21.614689 4037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:21.614733 4037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:21.614769 4037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:21.614794 4037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:21.621891 4037 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33537
I20260619 16:49:21.621964 4165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33537 every 8 connection(s)
I20260619 16:49:21.622372 4037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:49:21.628851 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 4037
I20260619 16:49:21.628997 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:49:21.629236 392 external_mini_cluster.cc:1428] Setting key be4bcad9e5caec68063dc647656f11dc
Jun 19 16:49:21 dist-test-slave-jsw9 krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:21.637611 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:21.623338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59089 (local address 127.0.98.62:43055)
0619 16:49:21.623417 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:21.623421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:21.624023 (+ 602us) server_negotiation.cc:407] Connection header received
0619 16:49:21.625195 (+ 1172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:21.625197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:21.625214 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:21.625308 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:21.626404 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:21.627088 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:21.628802 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:21.629205 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:21.629310 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:21.632870 (+ 3560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:21.632873 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:21.632873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:21.632884 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:21.635611 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:21.636314 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:21.636316 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:21.636316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:21.636392 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:21.636747 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:21.636748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:21.636749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:21.636901 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:21.636970 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:21.637403 (+ 433us) server_negotiation.cc:299] Negotiation successful
0619 16:49:21.637510 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 16:49:21.637887 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:21.623296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43055 (local address 127.0.98.1:59089)
0619 16:49:21.623767 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:21.623858 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:21.625046 (+ 1188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:21.625482 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:21.625489 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:21.625868 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:21.626142 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:21.626149 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:21.627266 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:21.627267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:21.628635 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:21.628641 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:21.629362 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:21.629363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:21.629562 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:21.630351 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:21.630362 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:21.632726 (+ 2364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:21.635732 (+ 3006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:21.635751 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:21.635756 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:21.636208 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:21.636469 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:21.636470 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:21.636470 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:21.636653 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:21.636979 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:21.636987 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:21.637218 (+ 231us) client_negotiation.cc:769] Sending connection context
0619 16:49:21.637357 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 16:49:21.637591 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":102,"threads_started":1}
I20260619 16:49:21.638501 4166 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43055
I20260619 16:49:21.638631 4166 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:21.638862 4166 heartbeater.cc:511] Master 127.0.98.62:43055 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 16:49:21.639521 3975 ts_manager.cc:194] Registered new tserver with Master: a00c27b4d19f4d829be84a4cc2a07f58 (127.0.98.1:33537)
I20260619 16:49:21.640200 3975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59089
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:49:21.664139 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43055
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:21.767401 4171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:21.767699 4171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:21.767731 4171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:21.769184 4171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:21.769241 4171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:21.769273 4171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:49:21.771422 4171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:21.771761 4171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:21.772015 4171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:21.778497 4179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:21.778497 4176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:21.778501 4177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:21.778925 4171 server_base.cc:1034] running on GCE node
I20260619 16:49:21.779176 4171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:21.779675 4171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:21.780855 4171 hybrid_clock.cc:648] HybridClock initialized: now 1781887761780840 us; error 36 us; skew 500 ppm
Jun 19 16:49:21 dist-test-slave-jsw9 krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:21.784309 4171 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:49:21.785181 4171 webserver.cc:460] Webserver started at http://127.0.98.2:40105/ using document root <none> and password file <none>
I20260619 16:49:21.785491 4171 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:21.785709 4171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:21.785923 4171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:21.787361 4171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-1/data/instance:
uuid: "17d6138c91534e989f617577c810a9cf"
format_stamp: "Formatted at 2026-06-19 16:49:21 on dist-test-slave-jsw9"
server_key: "1c0ccd3f3808d0f912097b88821d1b86"
server_key_iv: "0e262f8bc1f520f1a20d690d9e52e261"
server_key_version: "encryptionkey@0"
I20260619 16:49:21.787786 4171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-1/wal/instance:
uuid: "17d6138c91534e989f617577c810a9cf"
format_stamp: "Formatted at 2026-06-19 16:49:21 on dist-test-slave-jsw9"
server_key: "1c0ccd3f3808d0f912097b88821d1b86"
server_key_iv: "0e262f8bc1f520f1a20d690d9e52e261"
server_key_version: "encryptionkey@0"
I20260619 16:49:21.789563 4171 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:49:21.790578 4186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:21.790781 4171 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:21.790894 4171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-1/wal
uuid: "17d6138c91534e989f617577c810a9cf"
format_stamp: "Formatted at 2026-06-19 16:49:21 on dist-test-slave-jsw9"
server_key: "1c0ccd3f3808d0f912097b88821d1b86"
server_key_iv: "0e262f8bc1f520f1a20d690d9e52e261"
server_key_version: "encryptionkey@0"
I20260619 16:49:21.790990 4171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:21.802142 4171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:21.804950 4171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:21.805105 4171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:21.805364 4171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:21.805709 4171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:21.805742 4171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:21.805768 4171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:21.805779 4171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:21.812800 4171 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35715
I20260619 16:49:21.812865 4299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35715 every 8 connection(s)
I20260619 16:49:21.813234 4171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 16:49:21 dist-test-slave-jsw9 krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:21.821540 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 4171
I20260619 16:49:21.821655 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:49:21.821866 392 external_mini_cluster.cc:1428] Setting key 3626e7151222fad3382351a2a83731ac
I20260619 16:49:21.825258 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:21.814332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52775 (local address 127.0.98.62:43055)
0619 16:49:21.814429 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:21.814432 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:21.814677 (+ 245us) server_negotiation.cc:407] Connection header received
0619 16:49:21.815911 (+ 1234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:21.815913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:21.815930 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:21.815989 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:21.816817 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:21.817361 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:21.818606 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:21.818818 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:21.818879 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:21.821730 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:21.821732 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:21.821732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:21.821741 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:21.823372 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:21.823938 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:21.823940 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:21.823940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:21.824007 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:21.824335 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:21.824336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:21.824337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:21.824519 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:21.824638 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:21.825040 (+ 402us) server_negotiation.cc:299] Negotiation successful
0619 16:49:21.825160 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:49:21.825505 4302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:21.814158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43055 (local address 127.0.98.2:52775)
0619 16:49:21.814389 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:21.814451 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:21.815783 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:21.816081 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:21.816092 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:21.816452 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:21.816711 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:21.816717 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:21.817460 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:21.817461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:21.818495 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:21.818499 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:21.818898 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:21.818898 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:21.819033 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:21.819803 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:21.819814 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:21.821580 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:21.823480 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:21.823494 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:21.823498 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:21.823856 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:21.824067 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:21.824068 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:21.824069 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:21.824227 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:21.824588 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:21.824595 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:21.824836 (+ 241us) client_negotiation.cc:769] Sending connection context
0619 16:49:21.824982 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 16:49:21.825215 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":125,"threads_started":1}
I20260619 16:49:21.826054 4300 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43055
I20260619 16:49:21.826220 4300 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:21.826459 4300 heartbeater.cc:511] Master 127.0.98.62:43055 requested a full tablet report, sending...
I20260619 16:49:21.826969 3975 ts_manager.cc:194] Registered new tserver with Master: 17d6138c91534e989f617577c810a9cf (127.0.98.2:35715)
I20260619 16:49:21.827409 3975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52775
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:49:21.855028 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43055
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:21.956773 4305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:21.957065 4305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:21.957096 4305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:21.958633 4305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:21.958694 4305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:21.958730 4305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:49:21.960595 4305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:21.960868 4305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:21.961112 4305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:21.967005 4313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:21.966981 4311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:21.967200 4305 server_base.cc:1034] running on GCE node
W20260619 16:49:21.966981 4310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:21.967489 4305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:21.967805 4305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:21.969002 4305 hybrid_clock.cc:648] HybridClock initialized: now 1781887761968983 us; error 38 us; skew 500 ppm
Jun 19 16:49:21 dist-test-slave-jsw9 krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:21.972304 4305 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:49:21.972908 4305 webserver.cc:460] Webserver started at http://127.0.98.3:43039/ using document root <none> and password file <none>
I20260619 16:49:21.973155 4305 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:21.973217 4305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:21.973356 4305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:21.974711 4305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-2/data/instance:
uuid: "7369f40cf5a342ecb8f6fea0d4c184d0"
format_stamp: "Formatted at 2026-06-19 16:49:21 on dist-test-slave-jsw9"
server_key: "fb2834c8fac95b934d2cdbcaffb35636"
server_key_iv: "28c053090520e9acc34e177aac705258"
server_key_version: "encryptionkey@0"
I20260619 16:49:21.975097 4305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7369f40cf5a342ecb8f6fea0d4c184d0"
format_stamp: "Formatted at 2026-06-19 16:49:21 on dist-test-slave-jsw9"
server_key: "fb2834c8fac95b934d2cdbcaffb35636"
server_key_iv: "28c053090520e9acc34e177aac705258"
server_key_version: "encryptionkey@0"
I20260619 16:49:21.976635 4305 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 16:49:21.977535 4320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:21.977804 4305 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:21.977921 4305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-2/wal
uuid: "7369f40cf5a342ecb8f6fea0d4c184d0"
format_stamp: "Formatted at 2026-06-19 16:49:21 on dist-test-slave-jsw9"
server_key: "fb2834c8fac95b934d2cdbcaffb35636"
server_key_iv: "28c053090520e9acc34e177aac705258"
server_key_version: "encryptionkey@0"
I20260619 16:49:21.978003 4305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:21.995549 4305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:21.998786 4305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:21.998966 4305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:21.999284 4305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:21.999696 4305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:21.999738 4305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:21.999773 4305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:21.999790 4305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:22.006990 4305 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36033
I20260619 16:49:22.007088 4433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36033 every 8 connection(s)
I20260619 16:49:22.007472 4305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:49:22.012418 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 4305
I20260619 16:49:22.012531 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781887741903642-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:49:22.012742 392 external_mini_cluster.cc:1428] Setting key d1021ee2d0e371b96706f1e0d5997c1c
Jun 19 16:49:22 dist-test-slave-jsw9 krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:22.019740 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:22.008514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57365 (local address 127.0.98.62:43055)
0619 16:49:22.008580 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:22.008583 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:22.008859 (+ 276us) server_negotiation.cc:407] Connection header received
0619 16:49:22.009915 (+ 1056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:22.009917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:22.009930 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:22.010000 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:22.010921 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:22.011492 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:22.012710 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:22.012978 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:22.013073 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:22.015914 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:22.015916 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:22.015917 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:22.015930 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:22.017721 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:22.018495 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:22.018496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:22.018497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:22.018576 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:22.018893 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:22.018893 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:22.018894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:22.019041 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:22.019098 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:22.019546 (+ 448us) server_negotiation.cc:299] Negotiation successful
0619 16:49:22.019656 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 16:49:22.019968 4436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:22.008433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43055 (local address 127.0.98.3:57365)
0619 16:49:22.008697 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:22.008749 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:22.009784 (+ 1035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:22.010087 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:22.010091 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:22.010485 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:22.010787 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:22.010795 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:22.011607 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:22.011608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:22.012599 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:22.012605 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:22.013089 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:22.013091 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:22.013293 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:22.013959 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:22.013972 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:22.015653 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:22.017843 (+ 2190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:22.017855 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:22.017859 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:22.018316 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:22.018659 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:22.018660 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:22.018660 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:22.018816 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:22.019121 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:22.019129 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:22.019330 (+ 201us) client_negotiation.cc:769] Sending connection context
0619 16:49:22.019443 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 16:49:22.019632 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":97,"threads_started":1}
I20260619 16:49:22.020848 4434 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43055
I20260619 16:49:22.021047 4434 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:22.021354 4434 heartbeater.cc:511] Master 127.0.98.62:43055 requested a full tablet report, sending...
I20260619 16:49:22.021895 3975 ts_manager.cc:194] Registered new tserver with Master: 7369f40cf5a342ecb8f6fea0d4c184d0 (127.0.98.3:36033)
I20260619 16:49:22.022531 3975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57365
I20260619 16:49:22.025162 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 16:49:22 dist-test-slave-jsw9 krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 16:49:22 dist-test-slave-jsw9 krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 16:49:22.051584 4441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:22.042419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59814 (local address 127.0.98.1:33537)
0619 16:49:22.042628 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:22.042634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:22.042644 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:49:22.042674 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:22.042676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:22.042693 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:22.042765 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:22.043413 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:22.044211 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:22.044997 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:22.045375 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:22.045441 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:22.048273 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:22.048286 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:22.048287 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:22.048296 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:22.050246 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:22.050618 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:22.050624 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:22.050624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:22.050707 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:22.050881 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:22.050883 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:22.050883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:22.051068 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:22.051250 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:22.051350 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 16:49:22.051451 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":113,"threads_started":1}
Jun 19 16:49:22 dist-test-slave-jsw9 krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:22.061200 4014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:22.053403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49672 (local address 127.0.98.62:43055)
0619 16:49:22.053497 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:22.053499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:22.053509 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:49:22.053540 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:22.053541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:22.053552 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:22.053602 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:22.054134 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:22.054724 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:22.055433 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:22.055635 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:22.055693 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:22.058430 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:22.058432 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:22.058432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:22.058447 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:22.059934 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:22.060328 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:22.060329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:22.060330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:22.060394 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:22.060571 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:22.060572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:22.060572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:22.060772 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:22.060921 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:22.060995 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 16:49:22.061086 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260619 16:49:22.061741 3975 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49672
I20260619 16:49:22.062361 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 4037
I20260619 16:49:22.068099 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 4171
I20260619 16:49:22.072919 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 4305
I20260619 16:49:22.077466 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 3943
2026-06-19T16:49:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2039 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:49:22 dist-test-slave-jsw9 krb5kdc[4446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:49:22 dist-test-slave-jsw9 krb5kdc[4446](info): set up 2 sockets
Jun 19 16:49:22 dist-test-slave-jsw9 krb5kdc[4446](info): commencing operation
krb5kdc: starting...
W20260619 16:49:23.102727 392 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:49:23 dist-test-slave-jsw9 krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:49:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:49:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:49:23.252028 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35559
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:23.356973 4459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:23.357267 4459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:23.357295 4459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:23.358757 4459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:23.358812 4459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:23.358826 4459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:23.358839 4459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:23.358851 4459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:49:23.361402 4459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:23.361975 4459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:23.362341 4459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:49:23.368065 4459 server_base.cc:1034] running on GCE node
W20260619 16:49:23.368026 4465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:23.368027 4464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:23.368221 4467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:23.368590 4459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:23.368981 4459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:23.370179 4459 hybrid_clock.cc:648] HybridClock initialized: now 1781887763370161 us; error 38 us; skew 500 ppm
Jun 19 16:49:23 dist-test-slave-jsw9 krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:23.377439 4459 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:49:23.377892 4459 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:23.377966 4459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:23.378118 4459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:23.379599 4459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "08e040830f02412ca278079c4f39a58e"
format_stamp: "Formatted at 2026-06-19 16:49:23 on dist-test-slave-jsw9"
server_key: "e24ed8a45ab18149d6a92236b928f843"
server_key_iv: "4c4aadf88f33dc2713a96a6e0e3c96f6"
server_key_version: "encryptionkey@0"
I20260619 16:49:23.379972 4459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "08e040830f02412ca278079c4f39a58e"
format_stamp: "Formatted at 2026-06-19 16:49:23 on dist-test-slave-jsw9"
server_key: "e24ed8a45ab18149d6a92236b928f843"
server_key_iv: "4c4aadf88f33dc2713a96a6e0e3c96f6"
server_key_version: "encryptionkey@0"
I20260619 16:49:23.381705 4459 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 16:49:23.382658 4473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:23.382862 4459 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 16:49:23.382952 4459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "08e040830f02412ca278079c4f39a58e"
format_stamp: "Formatted at 2026-06-19 16:49:23 on dist-test-slave-jsw9"
server_key: "e24ed8a45ab18149d6a92236b928f843"
server_key_iv: "4c4aadf88f33dc2713a96a6e0e3c96f6"
server_key_version: "encryptionkey@0"
I20260619 16:49:23.383033 4459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:23.393838 4459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:23.396457 4459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:23.396662 4459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:23.401816 4459 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35559
I20260619 16:49:23.401841 4525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35559 every 8 connection(s)
I20260619 16:49:23.402238 4459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:23.404629 4526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:23.407770 4526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08e040830f02412ca278079c4f39a58e: Bootstrap starting.
I20260619 16:49:23.408563 4526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08e040830f02412ca278079c4f39a58e: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:23.408926 4526 log.cc:824] T 00000000000000000000000000000000 P 08e040830f02412ca278079c4f39a58e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:23.408989 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 4459
I20260619 16:49:23.409093 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:49:23.409317 392 external_mini_cluster.cc:1428] Setting key c864f28e709bab63fc83081c9302d269
I20260619 16:49:23.409850 4526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08e040830f02412ca278079c4f39a58e: No bootstrap required, opened a new log
I20260619 16:49:23.412073 4526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08e040830f02412ca278079c4f39a58e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08e040830f02412ca278079c4f39a58e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35559 } }
I20260619 16:49:23.412314 4526 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08e040830f02412ca278079c4f39a58e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:23.412345 4526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08e040830f02412ca278079c4f39a58e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08e040830f02412ca278079c4f39a58e, State: Initialized, Role: FOLLOWER
I20260619 16:49:23.412508 4526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08e040830f02412ca278079c4f39a58e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08e040830f02412ca278079c4f39a58e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35559 } }
I20260619 16:49:23.412600 4526 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08e040830f02412ca278079c4f39a58e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:23.412631 4526 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08e040830f02412ca278079c4f39a58e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:23.412674 4526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08e040830f02412ca278079c4f39a58e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:23.413823 4526 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08e040830f02412ca278079c4f39a58e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08e040830f02412ca278079c4f39a58e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35559 } }
I20260619 16:49:23.413969 4526 leader_election.cc:302] T 00000000000000000000000000000000 P 08e040830f02412ca278079c4f39a58e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08e040830f02412ca278079c4f39a58e; no voters:
I20260619 16:49:23.414208 4526 leader_election.cc:288] T 00000000000000000000000000000000 P 08e040830f02412ca278079c4f39a58e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:49:23.414357 4531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08e040830f02412ca278079c4f39a58e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:23.414541 4531 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08e040830f02412ca278079c4f39a58e [term 1 LEADER]: Becoming Leader. State: Replica: 08e040830f02412ca278079c4f39a58e, State: Running, Role: LEADER
I20260619 16:49:23.414585 4526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08e040830f02412ca278079c4f39a58e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:23.415434 4531 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08e040830f02412ca278079c4f39a58e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08e040830f02412ca278079c4f39a58e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35559 } }
I20260619 16:49:23.416149 4532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08e040830f02412ca278079c4f39a58e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08e040830f02412ca278079c4f39a58e. Latest consensus state: current_term: 1 leader_uuid: "08e040830f02412ca278079c4f39a58e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08e040830f02412ca278079c4f39a58e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35559 } } }
I20260619 16:49:23.416265 4533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08e040830f02412ca278079c4f39a58e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08e040830f02412ca278079c4f39a58e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08e040830f02412ca278079c4f39a58e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35559 } } }
I20260619 16:49:23.416491 4532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08e040830f02412ca278079c4f39a58e [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:23.416494 4533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08e040830f02412ca278079c4f39a58e [sys.catalog]: This master's current role is: LEADER
W20260619 16:49:23.416857 4545 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 08e040830f02412ca278079c4f39a58e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:49:23.416913 4545 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:49:23.417014 4548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:23.417232 4548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:23.419319 4548 catalog_manager.cc:1347] Generated new cluster ID: 50812afeb64746d29c7ec426a683e218
I20260619 16:49:23.419386 4548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 16:49:23 dist-test-slave-jsw9 krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:23.427973 4530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:23.410032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53254 (local address 127.0.98.62:35559)
0619 16:49:23.410258 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:23.410268 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:23.410298 (+ 30us) server_negotiation.cc:407] Connection header received
0619 16:49:23.412120 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:23.412137 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:23.412252 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:23.412713 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:23.413770 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:23.415167 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:23.416651 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:23.417363 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:23.417871 (+ 508us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:23.420994 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:23.421001 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:23.421001 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:23.421017 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:23.424759 (+ 3742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:23.425627 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:23.425640 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:23.425641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:23.425957 (+ 316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:23.426428 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:23.426429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:23.426430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:23.426892 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:23.427072 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:23.427220 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 16:49:23.427402 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":80,"threads_started":1}
I20260619 16:49:23.431644 4548 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:49:23.432369 4548 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:23.439054 4548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08e040830f02412ca278079c4f39a58e: Generated new TSK 0
I20260619 16:49:23.439442 4548 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:49:23.479779 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35559
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:23.594869 4552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:23.595163 4552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:23.595196 4552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:23.596724 4552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:23.596782 4552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:23.596817 4552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:49:23.598788 4552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:23.599134 4552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:23.599409 4552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:49:23.605266 4552 server_base.cc:1034] running on GCE node
W20260619 16:49:23.605314 4557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:23.605202 4558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:23.605398 4560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:23.605667 4552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:23.605999 4552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:23.607153 4552 hybrid_clock.cc:648] HybridClock initialized: now 1781887763607129 us; error 41 us; skew 500 ppm
Jun 19 16:49:23 dist-test-slave-jsw9 krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:23.609972 4552 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:49:23.610411 4552 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:23.610471 4552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:23.610613 4552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:23.611857 4552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-0/data/instance:
uuid: "854590798e4c470e92bb2bfdf10e4126"
format_stamp: "Formatted at 2026-06-19 16:49:23 on dist-test-slave-jsw9"
server_key: "0ec7bb65100a860ed328bb55a73f2b74"
server_key_iv: "fa9c92f5524be0e41d12fca23f39ecb5"
server_key_version: "encryptionkey@0"
I20260619 16:49:23.612226 4552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-0/wal/instance:
uuid: "854590798e4c470e92bb2bfdf10e4126"
format_stamp: "Formatted at 2026-06-19 16:49:23 on dist-test-slave-jsw9"
server_key: "0ec7bb65100a860ed328bb55a73f2b74"
server_key_iv: "fa9c92f5524be0e41d12fca23f39ecb5"
server_key_version: "encryptionkey@0"
I20260619 16:49:23.613634 4552 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 16:49:23.614415 4566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:23.614558 4552 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 16:49:23.614651 4552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-0/wal
uuid: "854590798e4c470e92bb2bfdf10e4126"
format_stamp: "Formatted at 2026-06-19 16:49:23 on dist-test-slave-jsw9"
server_key: "0ec7bb65100a860ed328bb55a73f2b74"
server_key_iv: "fa9c92f5524be0e41d12fca23f39ecb5"
server_key_version: "encryptionkey@0"
I20260619 16:49:23.614734 4552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:23.632604 4552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:23.635138 4552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:23.635623 4552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:23.635939 4552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:23.636322 4552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:23.636358 4552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:23.636392 4552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:23.636417 4552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:23.643858 4552 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45849
I20260619 16:49:23.643935 4679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45849 every 8 connection(s)
I20260619 16:49:23.644255 4552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:49:23.647207 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 4552
I20260619 16:49:23.647370 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:49:23.647580 392 external_mini_cluster.cc:1428] Setting key 24ed914f3a20ac24f902917f8d15015e
Jun 19 16:49:23 dist-test-slave-jsw9 krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 16:49:23.657958 4530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:23.645242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48295 (local address 127.0.98.62:35559)
0619 16:49:23.645458 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:23.645461 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:23.645660 (+ 199us) server_negotiation.cc:407] Connection header received
0619 16:49:23.646323 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:23.646325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:23.646337 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:23.646396 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:23.647876 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:23.648510 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:23.649667 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:23.649896 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:23.649958 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:23.652982 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:23.652985 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:23.652985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:23.652995 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:23.655195 (+ 2200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:23.655910 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:23.655912 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:23.655912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:23.655985 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:23.656590 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:23.656592 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:23.656593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:23.656791 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:23.656875 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:23.657745 (+ 870us) server_negotiation.cc:299] Negotiation successful
0619 16:49:23.657867 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165}
I20260619 16:49:23.658257 4682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:23.645208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35559 (local address 127.0.98.1:48295)
0619 16:49:23.645521 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:23.645566 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:23.646149 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:23.646603 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:23.646608 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:23.646971 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:23.647748 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:23.647758 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:23.648611 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:23.648612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:23.649568 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:23.649571 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:23.649979 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:23.649980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:23.650129 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:23.650773 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:23.650784 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:23.652775 (+ 1991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:23.655358 (+ 2583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:23.655375 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:23.655379 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:23.655798 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:23.656065 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:23.656066 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:23.656067 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:23.656283 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:23.656887 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:23.656897 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:23.657137 (+ 240us) client_negotiation.cc:769] Sending connection context
0619 16:49:23.657635 (+ 498us) client_negotiation.cc:240] Negotiation successful
0619 16:49:23.657941 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":202,"threads_started":1}
I20260619 16:49:23.658977 4680 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35559
I20260619 16:49:23.659098 4680 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:23.659264 4680 heartbeater.cc:511] Master 127.0.98.62:35559 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 16:49:23.664474 4485 ts_manager.cc:194] Registered new tserver with Master: 854590798e4c470e92bb2bfdf10e4126 (127.0.98.1:45849)
I20260619 16:49:23.665885 4485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48295
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:49:23.695976 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35559
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:23.798660 4685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:23.798954 4685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:23.798985 4685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:23.800400 4685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:23.800443 4685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:23.800478 4685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:49:23.802291 4685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:23.802601 4685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:23.802850 4685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:23.809021 4691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:23.809007 4693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:23.809289 4685 server_base.cc:1034] running on GCE node
W20260619 16:49:23.809007 4690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:23.809602 4685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:23.809924 4685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:23.811092 4685 hybrid_clock.cc:648] HybridClock initialized: now 1781887763811061 us; error 47 us; skew 500 ppm
Jun 19 16:49:23 dist-test-slave-jsw9 krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:23.814531 4685 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:49:23.815186 4685 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:23.815271 4685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:23.815690 4685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:23.817114 4685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-1/data/instance:
uuid: "999bb61423604c80b502ab3cd2cd5769"
format_stamp: "Formatted at 2026-06-19 16:49:23 on dist-test-slave-jsw9"
server_key: "dd92e9e913c605d95b6ca895555889fb"
server_key_iv: "1d7e4491d6e1cba393caf916c6d1c847"
server_key_version: "encryptionkey@0"
I20260619 16:49:23.817559 4685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-1/wal/instance:
uuid: "999bb61423604c80b502ab3cd2cd5769"
format_stamp: "Formatted at 2026-06-19 16:49:23 on dist-test-slave-jsw9"
server_key: "dd92e9e913c605d95b6ca895555889fb"
server_key_iv: "1d7e4491d6e1cba393caf916c6d1c847"
server_key_version: "encryptionkey@0"
I20260619 16:49:23.819309 4685 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 16:49:23.820215 4699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:23.820396 4685 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:23.820493 4685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-1/wal
uuid: "999bb61423604c80b502ab3cd2cd5769"
format_stamp: "Formatted at 2026-06-19 16:49:23 on dist-test-slave-jsw9"
server_key: "dd92e9e913c605d95b6ca895555889fb"
server_key_iv: "1d7e4491d6e1cba393caf916c6d1c847"
server_key_version: "encryptionkey@0"
I20260619 16:49:23.820565 4685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:23.839111 4685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:23.841564 4685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:23.841732 4685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:23.842012 4685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:23.842377 4685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:23.842414 4685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:23.842447 4685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:23.842471 4685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:23.849579 4685 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35713
I20260619 16:49:23.849654 4812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35713 every 8 connection(s)
I20260619 16:49:23.849967 4685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:49:23.853498 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 4685
I20260619 16:49:23.853624 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:49:23.853854 392 external_mini_cluster.cc:1428] Setting key f7b8c3c339ec2ff3714682bf7f72a3d1
Jun 19 16:49:23 dist-test-slave-jsw9 krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:23.861891 4530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:23.850887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42249 (local address 127.0.98.62:35559)
0619 16:49:23.850949 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:23.850951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:23.851189 (+ 238us) server_negotiation.cc:407] Connection header received
0619 16:49:23.851787 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:23.851788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:23.851801 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:23.851867 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:23.852991 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:23.853547 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:23.855117 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:23.855347 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:23.855415 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:23.858443 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:23.858446 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:23.858446 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:23.858457 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:23.860224 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:23.860736 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:23.860737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:23.860738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:23.860812 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:23.861066 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:23.861067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:23.861067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:23.861249 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:23.861323 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:23.861657 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 16:49:23.861774 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 16:49:23.862063 4815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:23.850855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35559 (local address 127.0.98.2:42249)
0619 16:49:23.851033 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:23.851085 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:23.851668 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:23.851938 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:23.851942 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:23.852272 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:23.852876 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:23.852883 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:23.853689 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:23.853691 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:23.854989 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:23.854993 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:23.855442 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:23.855444 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:23.855620 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:23.856289 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:23.856304 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:23.858286 (+ 1982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:23.860349 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:23.860358 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:23.860362 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:23.860654 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:23.860880 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:23.860881 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:23.860881 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:23.861003 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:23.861336 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:23.861343 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:23.861513 (+ 170us) client_negotiation.cc:769] Sending connection context
0619 16:49:23.861645 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 16:49:23.861817 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":74,"threads_started":1}
I20260619 16:49:23.862640 4813 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35559
I20260619 16:49:23.862754 4813 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:23.862941 4813 heartbeater.cc:511] Master 127.0.98.62:35559 requested a full tablet report, sending...
I20260619 16:49:23.863359 4485 ts_manager.cc:194] Registered new tserver with Master: 999bb61423604c80b502ab3cd2cd5769 (127.0.98.2:35713)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 16:49:23.863816 4485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42249
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:49:23.887781 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35559
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:24.008204 4818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:24.008541 4818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:24.008575 4818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:24.010279 4818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:24.010358 4818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:24.010399 4818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:49:24.012573 4818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:24.012944 4818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:24.013238 4818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:24.021998 4823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:24.022024 4818 server_base.cc:1034] running on GCE node
W20260619 16:49:24.022023 4826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:24.021999 4824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:24.023032 4818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:24.023711 4818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:24.024924 4818 hybrid_clock.cc:648] HybridClock initialized: now 1781887764024871 us; error 76 us; skew 500 ppm
Jun 19 16:49:24 dist-test-slave-jsw9 krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:24.029212 4818 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:49:24.029767 4818 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:24.029841 4818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:24.030011 4818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:24.031538 4818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-2/data/instance:
uuid: "1d79cbb651c34bd0b6628ed23381b3ad"
format_stamp: "Formatted at 2026-06-19 16:49:24 on dist-test-slave-jsw9"
server_key: "95b2f501b786d728e199bf88c5444b27"
server_key_iv: "771bf0fd28c028e96f3d3c32ec65a82a"
server_key_version: "encryptionkey@0"
I20260619 16:49:24.032174 4818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1d79cbb651c34bd0b6628ed23381b3ad"
format_stamp: "Formatted at 2026-06-19 16:49:24 on dist-test-slave-jsw9"
server_key: "95b2f501b786d728e199bf88c5444b27"
server_key_iv: "771bf0fd28c028e96f3d3c32ec65a82a"
server_key_version: "encryptionkey@0"
I20260619 16:49:24.033941 4818 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:49:24.035465 4832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:24.035879 4818 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:24.036020 4818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-2/wal
uuid: "1d79cbb651c34bd0b6628ed23381b3ad"
format_stamp: "Formatted at 2026-06-19 16:49:24 on dist-test-slave-jsw9"
server_key: "95b2f501b786d728e199bf88c5444b27"
server_key_iv: "771bf0fd28c028e96f3d3c32ec65a82a"
server_key_version: "encryptionkey@0"
I20260619 16:49:24.036123 4818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:24.063935 4818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:24.068080 4818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:24.068279 4818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:24.068536 4818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:24.068939 4818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:24.068979 4818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:24.069007 4818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:24.069018 4818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:24.076706 4818 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43753
I20260619 16:49:24.076723 4945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43753 every 8 connection(s)
I20260619 16:49:24.077114 4818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 16:49:24 dist-test-slave-jsw9 krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:24.085682 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 4818
I20260619 16:49:24.085824 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:49:24.086021 392 external_mini_cluster.cc:1428] Setting key bf98df2b9dacfd02cbb395a2ef6e610d
I20260619 16:49:24.089382 4530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:24.078147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38907 (local address 127.0.98.62:35559)
0619 16:49:24.078257 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:24.078260 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:24.078454 (+ 194us) server_negotiation.cc:407] Connection header received
0619 16:49:24.079172 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:24.079174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:24.079189 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:24.079248 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:24.080734 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:24.081293 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:24.082767 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:24.082998 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:24.083074 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:24.086031 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:24.086034 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:24.086034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:24.086045 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:24.087788 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:24.088284 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:24.088285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:24.088286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:24.088355 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:24.088608 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:24.088609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:24.088609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:24.088755 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:24.088817 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:24.089149 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 16:49:24.089272 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:49:24.089618 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:24.078051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35559 (local address 127.0.98.3:38907)
0619 16:49:24.078297 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:24.078358 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:24.079025 (+ 667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:24.079423 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:24.079429 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:24.079882 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:24.080530 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:24.080539 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:24.081407 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:24.081408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:24.082607 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:24.082648 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:24.083085 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:24.083087 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:24.083262 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:24.084008 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:24.084021 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:24.085892 (+ 1871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:24.087885 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:24.087908 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:24.087912 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:24.088210 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:24.088414 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:24.088415 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:24.088415 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:24.088550 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:24.088815 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:24.088823 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:24.089009 (+ 186us) client_negotiation.cc:769] Sending connection context
0619 16:49:24.089126 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 16:49:24.089338 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":142,"threads_started":1}
I20260619 16:49:24.090243 4946 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35559
I20260619 16:49:24.090381 4946 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:24.090555 4946 heartbeater.cc:511] Master 127.0.98.62:35559 requested a full tablet report, sending...
I20260619 16:49:24.091073 4485 ts_manager.cc:194] Registered new tserver with Master: 1d79cbb651c34bd0b6628ed23381b3ad (127.0.98.3:43753)
I20260619 16:49:24.091639 4485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38907
I20260619 16:49:24.099154 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:49:24.112609 4530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:24.101800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53262 (local address 127.0.98.62:35559)
0619 16:49:24.101890 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:24.101892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:24.102451 (+ 559us) server_negotiation.cc:407] Connection header received
0619 16:49:24.103496 (+ 1045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:24.103498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:24.103512 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:24.103688 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:24.104948 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:24.105816 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:24.107319 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:24.107535 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:24.107605 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:24.109288 (+ 1683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:24.109290 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:24.109291 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:24.109300 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:24.111117 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:24.111554 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:24.111555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:24.111556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:24.111621 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:24.111890 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:24.111891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:24.111892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:24.112114 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:24.112218 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:24.112404 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 16:49:24.112514 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:49:24.115295 4485 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53262:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:49:24.117893 4485 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:49:24.126852 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:24.121444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36882 (local address 127.0.98.2:35713)
0619 16:49:24.121644 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:24.121652 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:24.121932 (+ 280us) server_negotiation.cc:407] Connection header received
0619 16:49:24.121988 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:24.121991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:24.122011 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:24.122094 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:24.122362 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:24.123736 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:24.125749 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:24.126382 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:24.126457 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:24.126537 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:24.126682 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 16:49:24.126744 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":102,"threads_started":1}
I20260619 16:49:24.126852 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:24.121270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45849 (local address 127.0.0.1:35154)
0619 16:49:24.121675 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:24.121731 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:24.121838 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:24.122160 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:24.122161 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:24.122163 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:24.122235 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:24.122236 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:24.123868 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:24.123869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:24.125401 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:24.125405 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:24.126572 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:24.126573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:24.126661 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:24.126667 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 16:49:24.126712 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 16:49:24.126762 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":28,"thread_start_us":66,"threads_started":1}
I20260619 16:49:24.126852 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:24.121272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35713 (local address 127.0.0.1:36882)
0619 16:49:24.121739 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:24.121748 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:24.121838 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:24.122095 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:24.122098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:24.122100 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:24.122232 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:24.122235 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:24.123867 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:24.123869 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:24.125666 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:24.125668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:24.126465 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:24.126466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:24.126567 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:24.126585 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 16:49:24.126655 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 16:49:24.126734 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":58,"thread_start_us":122,"threads_started":1}
I20260619 16:49:24.127059 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:24.121560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35154 (local address 127.0.98.1:45849)
0619 16:49:24.122002 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:24.122007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:24.122016 (+ 9us) server_negotiation.cc:407] Connection header received
0619 16:49:24.122054 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:24.122056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:24.122072 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:24.122151 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:24.122366 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:24.123737 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:24.125494 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:24.126393 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:24.126468 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:24.126537 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:24.126901 (+ 364us) server_negotiation.cc:299] Negotiation successful
0619 16:49:24.126958 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":95,"threads_started":1}
I20260619 16:49:24.127875 4614 tablet_service.cc:1467] Processing CreateTablet for tablet ef2cf4548d194cdf8beead256a6fc1c0 (DEFAULT_TABLE table=test-table [id=c400470df64b4be2a229ab3c8d40171e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:24.127914 4747 tablet_service.cc:1467] Processing CreateTablet for tablet ef2cf4548d194cdf8beead256a6fc1c0 (DEFAULT_TABLE table=test-table [id=c400470df64b4be2a229ab3c8d40171e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:24.128051 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:24.121504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43753 (local address 127.0.0.1:44982)
0619 16:49:24.121898 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:24.121909 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:24.121973 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:24.124307 (+ 2334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:24.124308 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:24.124311 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:24.124372 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:24.124374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:24.125318 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:24.125319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:24.126475 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:24.126478 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:24.127796 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:24.127797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:24.127867 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:24.127873 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 16:49:24.127920 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 16:49:24.127971 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":75,"threads_started":1}
I20260619 16:49:24.128304 4614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef2cf4548d194cdf8beead256a6fc1c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:24.128304 4747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef2cf4548d194cdf8beead256a6fc1c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:24.128536 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:24.121734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44982 (local address 127.0.98.3:43753)
0619 16:49:24.124100 (+ 2366us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:24.124105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:24.124117 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:49:24.124164 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:24.124167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:24.124185 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:24.124292 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:24.124433 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:24.125220 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:24.126571 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:24.127622 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:24.127696 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:24.127759 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:24.128388 (+ 629us) server_negotiation.cc:299] Negotiation successful
0619 16:49:24.128446 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2312,"thread_start_us":156,"threads_started":1}
I20260619 16:49:24.129415 4880 tablet_service.cc:1467] Processing CreateTablet for tablet ef2cf4548d194cdf8beead256a6fc1c0 (DEFAULT_TABLE table=test-table [id=c400470df64b4be2a229ab3c8d40171e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:24.129817 4880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef2cf4548d194cdf8beead256a6fc1c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:24.131577 4963 tablet_bootstrap.cc:492] T ef2cf4548d194cdf8beead256a6fc1c0 P 854590798e4c470e92bb2bfdf10e4126: Bootstrap starting.
I20260619 16:49:24.132141 4964 tablet_bootstrap.cc:492] T ef2cf4548d194cdf8beead256a6fc1c0 P 999bb61423604c80b502ab3cd2cd5769: Bootstrap starting.
I20260619 16:49:24.132340 4963 tablet_bootstrap.cc:654] T ef2cf4548d194cdf8beead256a6fc1c0 P 854590798e4c470e92bb2bfdf10e4126: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:24.132611 4963 log.cc:824] T ef2cf4548d194cdf8beead256a6fc1c0 P 854590798e4c470e92bb2bfdf10e4126: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:24.132822 4964 tablet_bootstrap.cc:654] T ef2cf4548d194cdf8beead256a6fc1c0 P 999bb61423604c80b502ab3cd2cd5769: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:24.133365 4963 tablet_bootstrap.cc:492] T ef2cf4548d194cdf8beead256a6fc1c0 P 854590798e4c470e92bb2bfdf10e4126: No bootstrap required, opened a new log
I20260619 16:49:24.133433 4963 ts_tablet_manager.cc:1397] T ef2cf4548d194cdf8beead256a6fc1c0 P 854590798e4c470e92bb2bfdf10e4126: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:49:24.133442 4964 log.cc:824] T ef2cf4548d194cdf8beead256a6fc1c0 P 999bb61423604c80b502ab3cd2cd5769: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:24.133810 4966 tablet_bootstrap.cc:492] T ef2cf4548d194cdf8beead256a6fc1c0 P 1d79cbb651c34bd0b6628ed23381b3ad: Bootstrap starting.
I20260619 16:49:24.134490 4966 tablet_bootstrap.cc:654] T ef2cf4548d194cdf8beead256a6fc1c0 P 1d79cbb651c34bd0b6628ed23381b3ad: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:24.134516 4964 tablet_bootstrap.cc:492] T ef2cf4548d194cdf8beead256a6fc1c0 P 999bb61423604c80b502ab3cd2cd5769: No bootstrap required, opened a new log
I20260619 16:49:24.134574 4964 ts_tablet_manager.cc:1397] T ef2cf4548d194cdf8beead256a6fc1c0 P 999bb61423604c80b502ab3cd2cd5769: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:49:24.135193 4966 log.cc:824] T ef2cf4548d194cdf8beead256a6fc1c0 P 1d79cbb651c34bd0b6628ed23381b3ad: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:24.135470 4963 raft_consensus.cc:348] T ef2cf4548d194cdf8beead256a6fc1c0 P 854590798e4c470e92bb2bfdf10e4126 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999bb61423604c80b502ab3cd2cd5769" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35713 } } peers { permanent_uuid: "1d79cbb651c34bd0b6628ed23381b3ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43753 } } peers { permanent_uuid: "854590798e4c470e92bb2bfdf10e4126" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45849 } }
I20260619 16:49:24.135679 4963 raft_consensus.cc:374] T ef2cf4548d194cdf8beead256a6fc1c0 P 854590798e4c470e92bb2bfdf10e4126 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:24.135738 4963 raft_consensus.cc:729] T ef2cf4548d194cdf8beead256a6fc1c0 P 854590798e4c470e92bb2bfdf10e4126 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 854590798e4c470e92bb2bfdf10e4126, State: Initialized, Role: FOLLOWER
I20260619 16:49:24.135875 4963 consensus_queue.cc:260] T ef2cf4548d194cdf8beead256a6fc1c0 P 854590798e4c470e92bb2bfdf10e4126 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999bb61423604c80b502ab3cd2cd5769" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35713 } } peers { permanent_uuid: "1d79cbb651c34bd0b6628ed23381b3ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43753 } } peers { permanent_uuid: "854590798e4c470e92bb2bfdf10e4126" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45849 } }
I20260619 16:49:24.136183 4963 ts_tablet_manager.cc:1428] T ef2cf4548d194cdf8beead256a6fc1c0 P 854590798e4c470e92bb2bfdf10e4126: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 16:49:24.136308 4680 heartbeater.cc:503] Master 127.0.98.62:35559 was elected leader, sending a full tablet report...
I20260619 16:49:24.136348 4966 tablet_bootstrap.cc:492] T ef2cf4548d194cdf8beead256a6fc1c0 P 1d79cbb651c34bd0b6628ed23381b3ad: No bootstrap required, opened a new log
I20260619 16:49:24.136415 4966 ts_tablet_manager.cc:1397] T ef2cf4548d194cdf8beead256a6fc1c0 P 1d79cbb651c34bd0b6628ed23381b3ad: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 16:49:24.137128 4964 raft_consensus.cc:348] T ef2cf4548d194cdf8beead256a6fc1c0 P 999bb61423604c80b502ab3cd2cd5769 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999bb61423604c80b502ab3cd2cd5769" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35713 } } peers { permanent_uuid: "1d79cbb651c34bd0b6628ed23381b3ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43753 } } peers { permanent_uuid: "854590798e4c470e92bb2bfdf10e4126" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45849 } }
I20260619 16:49:24.137334 4964 raft_consensus.cc:374] T ef2cf4548d194cdf8beead256a6fc1c0 P 999bb61423604c80b502ab3cd2cd5769 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:24.137380 4964 raft_consensus.cc:729] T ef2cf4548d194cdf8beead256a6fc1c0 P 999bb61423604c80b502ab3cd2cd5769 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 999bb61423604c80b502ab3cd2cd5769, State: Initialized, Role: FOLLOWER
I20260619 16:49:24.137528 4964 consensus_queue.cc:260] T ef2cf4548d194cdf8beead256a6fc1c0 P 999bb61423604c80b502ab3cd2cd5769 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999bb61423604c80b502ab3cd2cd5769" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35713 } } peers { permanent_uuid: "1d79cbb651c34bd0b6628ed23381b3ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43753 } } peers { permanent_uuid: "854590798e4c470e92bb2bfdf10e4126" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45849 } }
I20260619 16:49:24.137852 4813 heartbeater.cc:503] Master 127.0.98.62:35559 was elected leader, sending a full tablet report...
I20260619 16:49:24.137866 4964 ts_tablet_manager.cc:1428] T ef2cf4548d194cdf8beead256a6fc1c0 P 999bb61423604c80b502ab3cd2cd5769: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260619 16:49:24.138739 4966 raft_consensus.cc:348] T ef2cf4548d194cdf8beead256a6fc1c0 P 1d79cbb651c34bd0b6628ed23381b3ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999bb61423604c80b502ab3cd2cd5769" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35713 } } peers { permanent_uuid: "1d79cbb651c34bd0b6628ed23381b3ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43753 } } peers { permanent_uuid: "854590798e4c470e92bb2bfdf10e4126" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45849 } }
I20260619 16:49:24.138926 4966 raft_consensus.cc:374] T ef2cf4548d194cdf8beead256a6fc1c0 P 1d79cbb651c34bd0b6628ed23381b3ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:24.138967 4966 raft_consensus.cc:729] T ef2cf4548d194cdf8beead256a6fc1c0 P 1d79cbb651c34bd0b6628ed23381b3ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d79cbb651c34bd0b6628ed23381b3ad, State: Initialized, Role: FOLLOWER
I20260619 16:49:24.139086 4966 consensus_queue.cc:260] T ef2cf4548d194cdf8beead256a6fc1c0 P 1d79cbb651c34bd0b6628ed23381b3ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999bb61423604c80b502ab3cd2cd5769" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35713 } } peers { permanent_uuid: "1d79cbb651c34bd0b6628ed23381b3ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43753 } } peers { permanent_uuid: "854590798e4c470e92bb2bfdf10e4126" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45849 } }
I20260619 16:49:24.139362 4946 heartbeater.cc:503] Master 127.0.98.62:35559 was elected leader, sending a full tablet report...
I20260619 16:49:24.139376 4966 ts_tablet_manager.cc:1428] T ef2cf4548d194cdf8beead256a6fc1c0 P 1d79cbb651c34bd0b6628ed23381b3ad: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
W20260619 16:49:24.145129 4681 tablet.cc:2369] T ef2cf4548d194cdf8beead256a6fc1c0 P 854590798e4c470e92bb2bfdf10e4126: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:49:24.227244 4970 raft_consensus.cc:482] T ef2cf4548d194cdf8beead256a6fc1c0 P 999bb61423604c80b502ab3cd2cd5769 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:24.227420 4970 raft_consensus.cc:504] T ef2cf4548d194cdf8beead256a6fc1c0 P 999bb61423604c80b502ab3cd2cd5769 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999bb61423604c80b502ab3cd2cd5769" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35713 } } peers { permanent_uuid: "1d79cbb651c34bd0b6628ed23381b3ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43753 } } peers { permanent_uuid: "854590798e4c470e92bb2bfdf10e4126" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45849 } }
I20260619 16:49:24.227946 4970 leader_election.cc:288] T ef2cf4548d194cdf8beead256a6fc1c0 P 999bb61423604c80b502ab3cd2cd5769 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d79cbb651c34bd0b6628ed23381b3ad (127.0.98.3:43753), 854590798e4c470e92bb2bfdf10e4126 (127.0.98.1:45849)
I20260619 16:49:24.231530 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:24.228262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33525 (local address 127.0.98.3:43753)
0619 16:49:24.228316 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:24.228318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:24.228349 (+ 31us) server_negotiation.cc:407] Connection header received
0619 16:49:24.228385 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:24.228386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:24.228399 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:24.228447 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:24.228627 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:24.229312 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:24.230651 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:24.231111 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:24.231188 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:24.231205 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:24.231386 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 16:49:24.231436 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 16:49:24.231582 4815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:24.228168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43753 (local address 127.0.98.2:33525)
0619 16:49:24.228226 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:24.228243 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:24.228311 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:24.228462 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:24.228463 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:24.228465 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:24.228555 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:24.228558 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:24.229566 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:24.229568 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:24.230527 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:24.230532 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:24.231211 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:24.231212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:24.231316 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:24.231320 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:49:24.231370 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 16:49:24.231429 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 16:49:24.231564 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:24.228356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36151 (local address 127.0.98.1:45849)
0619 16:49:24.228419 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:24.228421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:24.228542 (+ 121us) server_negotiation.cc:407] Connection header received
0619 16:49:24.228613 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:24.228614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:24.228629 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:24.228684 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:24.228913 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:24.229449 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:24.230738 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:24.231172 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:24.231250 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:24.231275 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:24.231432 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 16:49:24.231480 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 16:49:24.231602 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:24.228231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45849 (local address 127.0.98.2:36151)
0619 16:49:24.228454 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:24.228465 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:24.228535 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:24.228698 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:24.228699 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:24.228701 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:24.228784 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:24.228787 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:24.229699 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:24.229701 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:24.230629 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:24.230633 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:24.231276 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:24.231277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:24.231357 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:24.231361 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:49:24.231419 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 16:49:24.231469 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":166,"threads_started":1}
I20260619 16:49:24.232071 4900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef2cf4548d194cdf8beead256a6fc1c0" candidate_uuid: "999bb61423604c80b502ab3cd2cd5769" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d79cbb651c34bd0b6628ed23381b3ad" is_pre_election: true
I20260619 16:49:24.232251 4633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef2cf4548d194cdf8beead256a6fc1c0" candidate_uuid: "999bb61423604c80b502ab3cd2cd5769" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "854590798e4c470e92bb2bfdf10e4126" is_pre_election: true
I20260619 16:49:24.232326 4900 raft_consensus.cc:2454] T ef2cf4548d194cdf8beead256a6fc1c0 P 1d79cbb651c34bd0b6628ed23381b3ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 999bb61423604c80b502ab3cd2cd5769 in term 0.
I20260619 16:49:24.232414 4633 raft_consensus.cc:2454] T ef2cf4548d194cdf8beead256a6fc1c0 P 854590798e4c470e92bb2bfdf10e4126 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 999bb61423604c80b502ab3cd2cd5769 in term 0.
I20260619 16:49:24.232666 4700 leader_election.cc:302] T ef2cf4548d194cdf8beead256a6fc1c0 P 999bb61423604c80b502ab3cd2cd5769 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1d79cbb651c34bd0b6628ed23381b3ad, 854590798e4c470e92bb2bfdf10e4126, 999bb61423604c80b502ab3cd2cd5769; no voters:
I20260619 16:49:24.232818 4970 raft_consensus.cc:2781] T ef2cf4548d194cdf8beead256a6fc1c0 P 999bb61423604c80b502ab3cd2cd5769 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:49:24.232882 4970 raft_consensus.cc:482] T ef2cf4548d194cdf8beead256a6fc1c0 P 999bb61423604c80b502ab3cd2cd5769 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:49:24.232913 4970 raft_consensus.cc:3037] T ef2cf4548d194cdf8beead256a6fc1c0 P 999bb61423604c80b502ab3cd2cd5769 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:24.233764 4970 raft_consensus.cc:504] T ef2cf4548d194cdf8beead256a6fc1c0 P 999bb61423604c80b502ab3cd2cd5769 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999bb61423604c80b502ab3cd2cd5769" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35713 } } peers { permanent_uuid: "1d79cbb651c34bd0b6628ed23381b3ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43753 } } peers { permanent_uuid: "854590798e4c470e92bb2bfdf10e4126" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45849 } }
I20260619 16:49:24.233937 4970 leader_election.cc:288] T ef2cf4548d194cdf8beead256a6fc1c0 P 999bb61423604c80b502ab3cd2cd5769 [CANDIDATE]: Term 1 election: Requested vote from peers 1d79cbb651c34bd0b6628ed23381b3ad (127.0.98.3:43753), 854590798e4c470e92bb2bfdf10e4126 (127.0.98.1:45849)
I20260619 16:49:24.234259 4633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef2cf4548d194cdf8beead256a6fc1c0" candidate_uuid: "999bb61423604c80b502ab3cd2cd5769" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "854590798e4c470e92bb2bfdf10e4126"
I20260619 16:49:24.234251 4900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef2cf4548d194cdf8beead256a6fc1c0" candidate_uuid: "999bb61423604c80b502ab3cd2cd5769" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d79cbb651c34bd0b6628ed23381b3ad"
I20260619 16:49:24.234372 4900 raft_consensus.cc:3037] T ef2cf4548d194cdf8beead256a6fc1c0 P 1d79cbb651c34bd0b6628ed23381b3ad [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:24.234372 4633 raft_consensus.cc:3037] T ef2cf4548d194cdf8beead256a6fc1c0 P 854590798e4c470e92bb2bfdf10e4126 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:24.235489 4633 raft_consensus.cc:2454] T ef2cf4548d194cdf8beead256a6fc1c0 P 854590798e4c470e92bb2bfdf10e4126 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 999bb61423604c80b502ab3cd2cd5769 in term 1.
I20260619 16:49:24.235488 4900 raft_consensus.cc:2454] T ef2cf4548d194cdf8beead256a6fc1c0 P 1d79cbb651c34bd0b6628ed23381b3ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 999bb61423604c80b502ab3cd2cd5769 in term 1.
I20260619 16:49:24.235921 4700 leader_election.cc:302] T ef2cf4548d194cdf8beead256a6fc1c0 P 999bb61423604c80b502ab3cd2cd5769 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 854590798e4c470e92bb2bfdf10e4126, 999bb61423604c80b502ab3cd2cd5769; no voters:
I20260619 16:49:24.236054 4970 raft_consensus.cc:2781] T ef2cf4548d194cdf8beead256a6fc1c0 P 999bb61423604c80b502ab3cd2cd5769 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:24.236389 4970 raft_consensus.cc:686] T ef2cf4548d194cdf8beead256a6fc1c0 P 999bb61423604c80b502ab3cd2cd5769 [term 1 LEADER]: Becoming Leader. State: Replica: 999bb61423604c80b502ab3cd2cd5769, State: Running, Role: LEADER
I20260619 16:49:24.236606 4970 consensus_queue.cc:237] T ef2cf4548d194cdf8beead256a6fc1c0 P 999bb61423604c80b502ab3cd2cd5769 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999bb61423604c80b502ab3cd2cd5769" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35713 } } peers { permanent_uuid: "1d79cbb651c34bd0b6628ed23381b3ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43753 } } peers { permanent_uuid: "854590798e4c470e92bb2bfdf10e4126" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45849 } }
I20260619 16:49:24.237946 4485 catalog_manager.cc:5495] T ef2cf4548d194cdf8beead256a6fc1c0 P 999bb61423604c80b502ab3cd2cd5769 reported cstate change: term changed from 0 to 1, leader changed from <none> to 999bb61423604c80b502ab3cd2cd5769 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "999bb61423604c80b502ab3cd2cd5769" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "999bb61423604c80b502ab3cd2cd5769" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35713 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d79cbb651c34bd0b6628ed23381b3ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "854590798e4c470e92bb2bfdf10e4126" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45849 } health_report { overall_health: UNKNOWN } } }
I20260619 16:49:24.252121 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:24.248397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36884 (local address 127.0.98.2:35713)
0619 16:49:24.248532 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:24.248535 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:24.248544 (+ 9us) server_negotiation.cc:407] Connection header received
0619 16:49:24.248587 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:24.248588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:24.248612 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:24.248688 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:24.248849 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:24.249385 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:24.250759 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:24.250985 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:24.251184 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:24.251429 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:24.251590 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:24.251846 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:24.251950 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 16:49:24.251983 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260619 16:49:24.255260 4633 raft_consensus.cc:1261] T ef2cf4548d194cdf8beead256a6fc1c0 P 854590798e4c470e92bb2bfdf10e4126 [term 1 FOLLOWER]: Refusing update from remote peer 999bb61423604c80b502ab3cd2cd5769: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:24.255260 4900 raft_consensus.cc:1261] T ef2cf4548d194cdf8beead256a6fc1c0 P 1d79cbb651c34bd0b6628ed23381b3ad [term 1 FOLLOWER]: Refusing update from remote peer 999bb61423604c80b502ab3cd2cd5769: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:24.255870 4970 consensus_queue.cc:1035] T ef2cf4548d194cdf8beead256a6fc1c0 P 999bb61423604c80b502ab3cd2cd5769 [LEADER]: Connected to new peer: Peer: permanent_uuid: "854590798e4c470e92bb2bfdf10e4126" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:24.256110 4973 consensus_queue.cc:1035] T ef2cf4548d194cdf8beead256a6fc1c0 P 999bb61423604c80b502ab3cd2cd5769 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d79cbb651c34bd0b6628ed23381b3ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:24.259503 4978 mvcc.cc:205] Tried to move back new op lower bound from 7298612282385338368 to 7298612282314117120. Current Snapshot: MvccSnapshot[applied={T|T < 7298612282385338368}]
I20260619 16:49:24.260416 4977 mvcc.cc:205] Tried to move back new op lower bound from 7298612282385338368 to 7298612282314117120. Current Snapshot: MvccSnapshot[applied={T|T < 7298612282385338368}]
I20260619 16:49:24.262434 4490 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:24.262559 4490 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:24.263527 4490 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08e040830f02412ca278079c4f39a58e: Sending DeleteTablet for 3 replicas of tablet ef2cf4548d194cdf8beead256a6fc1c0
I20260619 16:49:24.263994 4614 tablet_service.cc:1514] Processing DeleteTablet for tablet ef2cf4548d194cdf8beead256a6fc1c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35154
I20260619 16:49:24.264123 4747 tablet_service.cc:1514] Processing DeleteTablet for tablet ef2cf4548d194cdf8beead256a6fc1c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36882
I20260619 16:49:24.264325 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 4552
I20260619 16:49:24.264446 4880 tablet_service.cc:1514] Processing DeleteTablet for tablet ef2cf4548d194cdf8beead256a6fc1c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44982
I20260619 16:49:24.264925 4987 tablet_replica.cc:330] T ef2cf4548d194cdf8beead256a6fc1c0 P 1d79cbb651c34bd0b6628ed23381b3ad: stopping tablet replica
I20260619 16:49:24.265058 4987 raft_consensus.cc:2229] T ef2cf4548d194cdf8beead256a6fc1c0 P 1d79cbb651c34bd0b6628ed23381b3ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:24.265192 4987 raft_consensus.cc:2258] T ef2cf4548d194cdf8beead256a6fc1c0 P 1d79cbb651c34bd0b6628ed23381b3ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:24.265399 4986 tablet_replica.cc:330] T ef2cf4548d194cdf8beead256a6fc1c0 P 999bb61423604c80b502ab3cd2cd5769: stopping tablet replica
I20260619 16:49:24.265689 4986 raft_consensus.cc:2229] T ef2cf4548d194cdf8beead256a6fc1c0 P 999bb61423604c80b502ab3cd2cd5769 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:49:24.266017 4986 raft_consensus.cc:2258] T ef2cf4548d194cdf8beead256a6fc1c0 P 999bb61423604c80b502ab3cd2cd5769 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:24.267076 4986 ts_tablet_manager.cc:1905] T ef2cf4548d194cdf8beead256a6fc1c0 P 999bb61423604c80b502ab3cd2cd5769: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:24.268656 4987 ts_tablet_manager.cc:1905] T ef2cf4548d194cdf8beead256a6fc1c0 P 1d79cbb651c34bd0b6628ed23381b3ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:24.270427 4986 ts_tablet_manager.cc:1918] T ef2cf4548d194cdf8beead256a6fc1c0 P 999bb61423604c80b502ab3cd2cd5769: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:24.270717 4986 log.cc:1196] T ef2cf4548d194cdf8beead256a6fc1c0 P 999bb61423604c80b502ab3cd2cd5769: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-1/wal/wals/ef2cf4548d194cdf8beead256a6fc1c0
I20260619 16:49:24.270776 4987 ts_tablet_manager.cc:1918] T ef2cf4548d194cdf8beead256a6fc1c0 P 1d79cbb651c34bd0b6628ed23381b3ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:24.270897 4987 log.cc:1196] T ef2cf4548d194cdf8beead256a6fc1c0 P 1d79cbb651c34bd0b6628ed23381b3ad: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781887741903642-392-0/minicluster-data/ts-2/wal/wals/ef2cf4548d194cdf8beead256a6fc1c0
I20260619 16:49:24.271297 4986 ts_tablet_manager.cc:1939] T ef2cf4548d194cdf8beead256a6fc1c0 P 999bb61423604c80b502ab3cd2cd5769: Deleting consensus metadata
I20260619 16:49:24.271302 4987 ts_tablet_manager.cc:1939] T ef2cf4548d194cdf8beead256a6fc1c0 P 1d79cbb651c34bd0b6628ed23381b3ad: Deleting consensus metadata
I20260619 16:49:24.271917 4477 catalog_manager.cc:4841] TS 1d79cbb651c34bd0b6628ed23381b3ad (127.0.98.3:43753): tablet ef2cf4548d194cdf8beead256a6fc1c0 (table test-table [id=c400470df64b4be2a229ab3c8d40171e]) successfully deleted
I20260619 16:49:24.272092 4477 catalog_manager.cc:4841] TS 999bb61423604c80b502ab3cd2cd5769 (127.0.98.2:35713): tablet ef2cf4548d194cdf8beead256a6fc1c0 (table test-table [id=c400470df64b4be2a229ab3c8d40171e]) successfully deleted
W20260619 16:49:24.273366 4474 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45849 (error 108)
I20260619 16:49:24.273804 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 4685
I20260619 16:49:24.275925 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:24.275714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45849 (local address 127.0.0.1:35168)
0619 16:49:24.275812 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:24.275869 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260619 16:49:24.276109 4474 catalog_manager.cc:4584] TS 854590798e4c470e92bb2bfdf10e4126 (127.0.98.1:45849): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ef2cf4548d194cdf8beead256a6fc1c0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45849: connect: Connection refused (error 111)
I20260619 16:49:24.279744 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 4818
I20260619 16:49:24.284732 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 4459
2026-06-19T16:49:24Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2208 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T16:49:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:49:24Z Disabled control of system clock
I20260619 16:49:24.315277 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34509
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 16:49:24.411446 4992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:24.411728 4992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:24.411754 4992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:24.413229 4992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:24.413295 4992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:24.413312 4992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:24.413324 4992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:24.413337 4992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:49:24.415990 4992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34509
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:24.416374 4992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:24.416917 4992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:24.422727 4997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:24.422727 5000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:24.422748 4998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:24.423053 4992 server_base.cc:1034] running on GCE node
I20260619 16:49:24.423326 4992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:24.423661 4992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:24.424875 4992 hybrid_clock.cc:648] HybridClock initialized: now 1781887764424852 us; error 42 us; skew 500 ppm
I20260619 16:49:24.426592 4992 webserver.cc:460] Webserver started at http://127.0.98.62:39847/ using document root <none> and password file <none>
I20260619 16:49:24.426920 4992 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:24.426975 4992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:24.427114 4992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:24.428385 4992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "7d7778a4c4ad40cd96e424008990d34e"
format_stamp: "Formatted at 2026-06-19 16:49:24 on dist-test-slave-jsw9"
server_key: "b2166abc20c095391b8dfa5802a269de"
server_key_iv: "c5e356d57b125e7309146b4da4acf2f1"
server_key_version: "encryptionkey@0"
I20260619 16:49:24.428771 4992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "7d7778a4c4ad40cd96e424008990d34e"
format_stamp: "Formatted at 2026-06-19 16:49:24 on dist-test-slave-jsw9"
server_key: "b2166abc20c095391b8dfa5802a269de"
server_key_iv: "c5e356d57b125e7309146b4da4acf2f1"
server_key_version: "encryptionkey@0"
I20260619 16:49:24.430328 4992 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260619 16:49:24.431330 5006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:24.431522 4992 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:24.431675 4992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "7d7778a4c4ad40cd96e424008990d34e"
format_stamp: "Formatted at 2026-06-19 16:49:24 on dist-test-slave-jsw9"
server_key: "b2166abc20c095391b8dfa5802a269de"
server_key_iv: "c5e356d57b125e7309146b4da4acf2f1"
server_key_version: "encryptionkey@0"
I20260619 16:49:24.431789 4992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:24.435936 4992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:24.436445 4992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:24.436622 4992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:24.441296 4992 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34509
I20260619 16:49:24.441375 5058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34509 every 8 connection(s)
I20260619 16:49:24.441758 4992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:24.442580 5059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:24.446092 5059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d7778a4c4ad40cd96e424008990d34e: Bootstrap starting.
I20260619 16:49:24.446909 5059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d7778a4c4ad40cd96e424008990d34e: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:24.447242 5059 log.cc:824] T 00000000000000000000000000000000 P 7d7778a4c4ad40cd96e424008990d34e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:24.448233 5059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d7778a4c4ad40cd96e424008990d34e: No bootstrap required, opened a new log
I20260619 16:49:24.450138 5059 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d7778a4c4ad40cd96e424008990d34e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7778a4c4ad40cd96e424008990d34e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34509 } }
I20260619 16:49:24.450402 5059 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d7778a4c4ad40cd96e424008990d34e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:24.450436 5059 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d7778a4c4ad40cd96e424008990d34e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d7778a4c4ad40cd96e424008990d34e, State: Initialized, Role: FOLLOWER
I20260619 16:49:24.450577 5059 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d7778a4c4ad40cd96e424008990d34e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7778a4c4ad40cd96e424008990d34e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34509 } }
I20260619 16:49:24.450639 5059 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d7778a4c4ad40cd96e424008990d34e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:24.450665 5059 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d7778a4c4ad40cd96e424008990d34e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:24.450700 5059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d7778a4c4ad40cd96e424008990d34e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:24.451269 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 4992
I20260619 16:49:24.451382 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:49:24.451535 5059 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d7778a4c4ad40cd96e424008990d34e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7778a4c4ad40cd96e424008990d34e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34509 } }
I20260619 16:49:24.451581 392 external_mini_cluster.cc:1428] Setting key 983c40960aeabf1331a7d072288843f4
I20260619 16:49:24.451670 5059 leader_election.cc:302] T 00000000000000000000000000000000 P 7d7778a4c4ad40cd96e424008990d34e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d7778a4c4ad40cd96e424008990d34e; no voters:
I20260619 16:49:24.451946 5059 leader_election.cc:288] T 00000000000000000000000000000000 P 7d7778a4c4ad40cd96e424008990d34e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:49:24.451985 5062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d7778a4c4ad40cd96e424008990d34e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:24.452118 5062 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d7778a4c4ad40cd96e424008990d34e [term 1 LEADER]: Becoming Leader. State: Replica: 7d7778a4c4ad40cd96e424008990d34e, State: Running, Role: LEADER
I20260619 16:49:24.452210 5062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d7778a4c4ad40cd96e424008990d34e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7778a4c4ad40cd96e424008990d34e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34509 } }
I20260619 16:49:24.452289 5059 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d7778a4c4ad40cd96e424008990d34e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:24.452600 5062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d7778a4c4ad40cd96e424008990d34e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d7778a4c4ad40cd96e424008990d34e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7778a4c4ad40cd96e424008990d34e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34509 } } }
I20260619 16:49:24.452677 5062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d7778a4c4ad40cd96e424008990d34e [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:24.452792 5062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d7778a4c4ad40cd96e424008990d34e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d7778a4c4ad40cd96e424008990d34e. Latest consensus state: current_term: 1 leader_uuid: "7d7778a4c4ad40cd96e424008990d34e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7778a4c4ad40cd96e424008990d34e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34509 } } }
I20260619 16:49:24.452847 5062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d7778a4c4ad40cd96e424008990d34e [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:24.453506 5069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:24.454561 5069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:24.454969 5066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:24.452271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50366 (local address 127.0.98.62:34509)
0619 16:49:24.452486 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:24.452497 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:24.453260 (+ 763us) server_negotiation.cc:407] Connection header received
0619 16:49:24.453496 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:24.453504 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:24.453547 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:24.453725 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:24.454463 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:24.454467 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:24.454468 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:24.454472 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:24.454518 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:24.454530 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:24.454592 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:24.454685 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 16:49:24.454766 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
I20260619 16:49:24.456416 5069 catalog_manager.cc:1347] Generated new cluster ID: 0b0ab65433b84d018c414ef5ada0f2fb
I20260619 16:49:24.456471 5069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:49:24.467423 5069 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:49:24.467689 5069 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:24.473052 5069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d7778a4c4ad40cd96e424008990d34e: Generated new TSK 0
I20260619 16:49:24.473399 5069 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:49:24.477284 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34509
--builtin_ntp_servers=127.0.98.20:42089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 16:49:24.582764 5083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:24.583156 5083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:24.584137 5083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:24.586597 5083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:24.586673 5083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:24.586723 5083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:49:24.589803 5083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:24.590248 5083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:24.590719 5083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:24.597563 5088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:24.597571 5089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:24.597694 5091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:24.597999 5083 server_base.cc:1034] running on GCE node
I20260619 16:49:24.598292 5083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:24.598600 5083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:24.599789 5083 hybrid_clock.cc:648] HybridClock initialized: now 1781887764599769 us; error 37 us; skew 500 ppm
I20260619 16:49:24.601377 5083 webserver.cc:460] Webserver started at http://127.0.98.1:42975/ using document root <none> and password file <none>
I20260619 16:49:24.601733 5083 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:24.601801 5083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:24.601950 5083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:24.603271 5083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-0/data/instance:
uuid: "8eed6745a89e450999dcc00e4c0454ef"
format_stamp: "Formatted at 2026-06-19 16:49:24 on dist-test-slave-jsw9"
server_key: "a9473ade3a36984f8ee49ef7241119f0"
server_key_iv: "4197819d386e06a99cc890cab53c79e7"
server_key_version: "encryptionkey@0"
I20260619 16:49:24.603683 5083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8eed6745a89e450999dcc00e4c0454ef"
format_stamp: "Formatted at 2026-06-19 16:49:24 on dist-test-slave-jsw9"
server_key: "a9473ade3a36984f8ee49ef7241119f0"
server_key_iv: "4197819d386e06a99cc890cab53c79e7"
server_key_version: "encryptionkey@0"
I20260619 16:49:24.605360 5083 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260619 16:49:24.606462 5097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:24.606676 5083 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:24.606794 5083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-0/wal
uuid: "8eed6745a89e450999dcc00e4c0454ef"
format_stamp: "Formatted at 2026-06-19 16:49:24 on dist-test-slave-jsw9"
server_key: "a9473ade3a36984f8ee49ef7241119f0"
server_key_iv: "4197819d386e06a99cc890cab53c79e7"
server_key_version: "encryptionkey@0"
I20260619 16:49:24.606899 5083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:24.612532 5083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:24.612910 5083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:24.613047 5083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:24.613274 5083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:24.613633 5083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:24.613668 5083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:24.613708 5083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:24.613730 5083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:24.622503 5083 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39659
I20260619 16:49:24.622560 5210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39659 every 8 connection(s)
I20260619 16:49:24.623068 5083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:49:24.624857 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 5083
I20260619 16:49:24.624986 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:49:24.625203 392 external_mini_cluster.cc:1428] Setting key 836d10f4101cb265a4ceb4dd0e3b33da
I20260619 16:49:24.626364 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34509
--builtin_ntp_servers=127.0.98.20:42089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 16:49:24.627249 5066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:24.624006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34227 (local address 127.0.98.62:34509)
0619 16:49:24.624085 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:24.624088 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:24.624374 (+ 286us) server_negotiation.cc:407] Connection header received
0619 16:49:24.625140 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:24.625141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:24.625151 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:24.625208 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:24.626971 (+ 1763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:24.626973 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:24.626973 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:24.626980 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:24.627004 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:24.627009 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:24.627082 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:24.627141 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 16:49:24.627160 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 16:49:24.627465 5213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:24.623965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34509 (local address 127.0.98.1:34227)
0619 16:49:24.624229 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:24.624273 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:24.624976 (+ 703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:24.625306 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:24.625311 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:24.625323 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:24.626779 (+ 1456us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:49:24.626780 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:24.626839 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:49:24.626878 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:24.627080 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:24.627083 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:24.627086 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:49:24.627151 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 16:49:24.627219 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":104,"threads_started":1}
I20260619 16:49:24.628069 5211 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34509
I20260619 16:49:24.628196 5211 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:24.628377 5211 heartbeater.cc:511] Master 127.0.98.62:34509 requested a full tablet report, sending...
I20260619 16:49:24.628957 5023 ts_manager.cc:194] Registered new tserver with Master: 8eed6745a89e450999dcc00e4c0454ef (127.0.98.1:39659)
W20260619 16:49:24.728267 5214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:24.728600 5214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:24.728641 5214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:24.730136 5214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:24.730255 5214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:24.730305 5214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:49:24.732869 5214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:24.733234 5214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:24.733659 5214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:49:24.740579 5214 server_base.cc:1034] running on GCE node
W20260619 16:49:24.740517 5222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:24.740518 5220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:24.740521 5219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:24.741072 5214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:24.741393 5214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:24.742636 5214 hybrid_clock.cc:648] HybridClock initialized: now 1781887764742603 us; error 46 us; skew 500 ppm
I20260619 16:49:24.748713 5214 webserver.cc:460] Webserver started at http://127.0.98.2:42997/ using document root <none> and password file <none>
I20260619 16:49:24.749082 5214 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:24.749146 5214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:24.749284 5214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:24.750697 5214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-1/data/instance:
uuid: "b648efed5499439da654b4c5625fbb5c"
format_stamp: "Formatted at 2026-06-19 16:49:24 on dist-test-slave-jsw9"
server_key: "8aa06f309c2b3bb883cac84d0af2b7e0"
server_key_iv: "6cde4194c34c066d2e7e9ddb014e4aac"
server_key_version: "encryptionkey@0"
I20260619 16:49:24.751104 5214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b648efed5499439da654b4c5625fbb5c"
format_stamp: "Formatted at 2026-06-19 16:49:24 on dist-test-slave-jsw9"
server_key: "8aa06f309c2b3bb883cac84d0af2b7e0"
server_key_iv: "6cde4194c34c066d2e7e9ddb014e4aac"
server_key_version: "encryptionkey@0"
I20260619 16:49:24.752993 5214 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:49:24.754029 5228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:24.754383 5214 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:24.754570 5214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-1/wal
uuid: "b648efed5499439da654b4c5625fbb5c"
format_stamp: "Formatted at 2026-06-19 16:49:24 on dist-test-slave-jsw9"
server_key: "8aa06f309c2b3bb883cac84d0af2b7e0"
server_key_iv: "6cde4194c34c066d2e7e9ddb014e4aac"
server_key_version: "encryptionkey@0"
I20260619 16:49:24.754686 5214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:24.759292 5214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:24.759645 5214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:24.759788 5214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:24.760039 5214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:24.760522 5214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:24.760592 5214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:24.760632 5214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:24.760648 5214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:24.768054 5214 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40739
I20260619 16:49:24.768102 5341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40739 every 8 connection(s)
I20260619 16:49:24.768502 5214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:49:24.772210 5066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:24.769353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42147 (local address 127.0.98.62:34509)
0619 16:49:24.769457 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:24.769460 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:24.769730 (+ 270us) server_negotiation.cc:407] Connection header received
0619 16:49:24.770432 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:24.770433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:24.770442 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:24.770500 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:24.771903 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:24.771905 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:24.771905 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:24.771912 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:24.771937 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:24.771942 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:24.771996 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:24.772091 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 16:49:24.772113 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:49:24.772475 5344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:24.769313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34509 (local address 127.0.98.2:42147)
0619 16:49:24.769548 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:24.769590 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:24.770312 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:24.770593 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:24.770599 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:24.770612 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:24.771693 (+ 1081us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:49:24.771693 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:24.771734 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:49:24.771776 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:24.772001 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:24.772007 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:24.772011 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:49:24.772101 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 16:49:24.772197 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":91,"threads_started":1}
I20260619 16:49:24.773370 5342 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34509
I20260619 16:49:24.773379 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 5214
I20260619 16:49:24.773518 5342 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:24.773522 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:49:24.773726 5342 heartbeater.cc:511] Master 127.0.98.62:34509 requested a full tablet report, sending...
I20260619 16:49:24.773749 392 external_mini_cluster.cc:1428] Setting key a08a451ab6011192a9e0e26720d89dca
I20260619 16:49:24.774264 5022 ts_manager.cc:194] Registered new tserver with Master: b648efed5499439da654b4c5625fbb5c (127.0.98.2:40739)
I20260619 16:49:24.775200 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34509
--builtin_ntp_servers=127.0.98.20:42089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 16:49:24.881981 5345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:24.882308 5345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:24.882337 5345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:24.883682 5345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:24.883724 5345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:24.883751 5345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:49:24.885175 5345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:24.885424 5345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:24.885739 5345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:24.891248 5351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:24.891242 5353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:24.891232 5350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:24.891646 5345 server_base.cc:1034] running on GCE node
I20260619 16:49:24.891882 5345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:24.892135 5345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:24.893307 5345 hybrid_clock.cc:648] HybridClock initialized: now 1781887764893291 us; error 31 us; skew 500 ppm
I20260619 16:49:24.895160 5345 webserver.cc:460] Webserver started at http://127.0.98.3:43733/ using document root <none> and password file <none>
I20260619 16:49:24.895818 5345 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:24.895913 5345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:24.896056 5345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:24.897545 5345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-2/data/instance:
uuid: "ff02e3ea75884f589eca57248bd136ad"
format_stamp: "Formatted at 2026-06-19 16:49:24 on dist-test-slave-jsw9"
server_key: "284ff577fbf532a592bd4fc9c649fce6"
server_key_iv: "89711f2d1ef981ee6f81790dd56432e1"
server_key_version: "encryptionkey@0"
I20260619 16:49:24.897994 5345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ff02e3ea75884f589eca57248bd136ad"
format_stamp: "Formatted at 2026-06-19 16:49:24 on dist-test-slave-jsw9"
server_key: "284ff577fbf532a592bd4fc9c649fce6"
server_key_iv: "89711f2d1ef981ee6f81790dd56432e1"
server_key_version: "encryptionkey@0"
I20260619 16:49:24.899919 5345 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.002s
I20260619 16:49:24.901047 5359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:24.901470 5345 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:24.901579 5345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-2/wal
uuid: "ff02e3ea75884f589eca57248bd136ad"
format_stamp: "Formatted at 2026-06-19 16:49:24 on dist-test-slave-jsw9"
server_key: "284ff577fbf532a592bd4fc9c649fce6"
server_key_iv: "89711f2d1ef981ee6f81790dd56432e1"
server_key_version: "encryptionkey@0"
I20260619 16:49:24.901678 5345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:24.905933 5345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:24.906342 5345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:24.906483 5345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:24.906735 5345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:24.907107 5345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:24.907141 5345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:24.907174 5345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:24.907198 5345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:24.915251 5345 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46773
I20260619 16:49:24.915335 5472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46773 every 8 connection(s)
I20260619 16:49:24.915693 5345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:49:24.919634 5066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:24.916542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43399 (local address 127.0.98.62:34509)
0619 16:49:24.916621 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:24.916623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:24.916903 (+ 280us) server_negotiation.cc:407] Connection header received
0619 16:49:24.917491 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:24.917493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:24.917500 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:24.917561 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:24.919339 (+ 1778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:24.919341 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:24.919342 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:24.919347 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:24.919365 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:24.919370 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:24.919426 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:24.919540 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 16:49:24.919558 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 16:49:24.919817 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:24.916511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34509 (local address 127.0.98.3:43399)
0619 16:49:24.916763 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:24.916805 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:24.917367 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:24.917710 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:24.917715 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:24.917726 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:24.919028 (+ 1302us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:49:24.919029 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:24.919141 (+ 112us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:49:24.919207 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:24.919430 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:24.919434 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:24.919437 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:49:24.919511 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 16:49:24.919575 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":116,"threads_started":1}
I20260619 16:49:24.920338 5473 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34509
I20260619 16:49:24.920454 5473 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:24.920624 5473 heartbeater.cc:511] Master 127.0.98.62:34509 requested a full tablet report, sending...
I20260619 16:49:24.921034 5023 ts_manager.cc:194] Registered new tserver with Master: ff02e3ea75884f589eca57248bd136ad (127.0.98.3:46773)
I20260619 16:49:24.921640 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 5345
I20260619 16:49:24.921758 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:49:24.921960 392 external_mini_cluster.cc:1428] Setting key 0265df5dd1df188fb89765e3ec63d6cc
I20260619 16:49:24.923019 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:49:24.925958 5066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:24.924789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50382 (local address 127.0.98.62:34509)
0619 16:49:24.924846 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:24.924848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:24.924905 (+ 57us) server_negotiation.cc:407] Connection header received
0619 16:49:24.924966 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:24.924967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:24.924973 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:24.925029 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:24.925676 (+ 647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:24.925677 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:24.925678 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:24.925682 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:24.925710 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:24.925715 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:24.925804 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:24.925875 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 16:49:24.925891 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 16:49:24.928157 5023 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50382:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:49:24.930071 5023 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:49:24.935348 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:24.933316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39659 (local address 127.0.0.1:32934)
0619 16:49:24.933861 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:24.933870 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:24.933914 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:24.934162 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:24.934169 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:24.934178 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:24.934852 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:49:24.934853 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:24.934874 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:49:24.934905 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:24.935121 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:24.935129 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:24.935132 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:49:24.935200 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 16:49:24.935216 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":66,"threads_started":1}
I20260619 16:49:24.935531 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:24.933290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46773 (local address 127.0.0.1:49148)
0619 16:49:24.933641 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:24.933694 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:24.933775 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:24.934444 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:24.934445 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:24.934448 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:24.935129 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:49:24.935129 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:24.935137 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:49:24.935150 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:24.935397 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:24.935398 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:24.935399 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 16:49:24.935444 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 16:49:24.935458 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":84,"threads_started":1}
I20260619 16:49:24.935750 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:24.933407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49148 (local address 127.0.98.3:46773)
0619 16:49:24.933872 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:24.933877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:24.933894 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:49:24.933926 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:24.933928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:24.933941 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:24.934017 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:24.935220 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:24.935230 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:24.935230 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:24.935235 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:24.935271 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:24.935321 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:24.935390 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:24.935634 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 16:49:24.935654 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":99,"threads_started":1}
I20260619 16:49:24.935861 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:24.933623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32934 (local address 127.0.98.1:39659)
0619 16:49:24.933791 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:24.933797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:24.933965 (+ 168us) server_negotiation.cc:407] Connection header received
0619 16:49:24.934002 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:24.934005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:24.934015 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:24.934087 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:24.934985 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:24.934994 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:24.934995 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:24.935000 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:24.935044 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:24.935066 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:24.935123 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:24.935676 (+ 553us) server_negotiation.cc:299] Negotiation successful
0619 16:49:24.935696 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
I20260619 16:49:24.935968 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:24.933302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40739 (local address 127.0.0.1:49406)
0619 16:49:24.933704 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:24.933714 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:24.933775 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:24.934963 (+ 1188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:24.934964 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:24.934967 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:24.935311 (+ 344us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:49:24.935311 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:24.935317 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:49:24.935326 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:24.935847 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:24.935848 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:24.935849 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 16:49:24.935891 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 16:49:24.935903 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":148,"threads_started":1}
I20260619 16:49:24.936254 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:24.933820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49406 (local address 127.0.98.2:40739)
0619 16:49:24.934720 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:24.934725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:24.934735 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:49:24.934809 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:24.934812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:24.934824 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:24.935243 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:24.935703 (+ 460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:24.935707 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:24.935707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:24.935710 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:24.935744 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:24.935758 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:24.936135 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:24.936159 (+ 24us) server_negotiation.cc:299] Negotiation successful
0619 16:49:24.936177 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":852,"thread_start_us":122,"threads_started":1}
I20260619 16:49:24.936720 5407 tablet_service.cc:1467] Processing CreateTablet for tablet 1bb9387dd8bd492080b51b45b504d996 (DEFAULT_TABLE table=test-table [id=05361f2aee664118bb1b5b30561c38c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:24.936913 5145 tablet_service.cc:1467] Processing CreateTablet for tablet 1bb9387dd8bd492080b51b45b504d996 (DEFAULT_TABLE table=test-table [id=05361f2aee664118bb1b5b30561c38c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:24.936913 5276 tablet_service.cc:1467] Processing CreateTablet for tablet 1bb9387dd8bd492080b51b45b504d996 (DEFAULT_TABLE table=test-table [id=05361f2aee664118bb1b5b30561c38c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:24.937148 5407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bb9387dd8bd492080b51b45b504d996. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:24.937282 5276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bb9387dd8bd492080b51b45b504d996. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:24.937296 5145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bb9387dd8bd492080b51b45b504d996. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:24.940922 5487 tablet_bootstrap.cc:492] T 1bb9387dd8bd492080b51b45b504d996 P ff02e3ea75884f589eca57248bd136ad: Bootstrap starting.
I20260619 16:49:24.941394 5489 tablet_bootstrap.cc:492] T 1bb9387dd8bd492080b51b45b504d996 P b648efed5499439da654b4c5625fbb5c: Bootstrap starting.
I20260619 16:49:24.941846 5487 tablet_bootstrap.cc:654] T 1bb9387dd8bd492080b51b45b504d996 P ff02e3ea75884f589eca57248bd136ad: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:24.942087 5489 tablet_bootstrap.cc:654] T 1bb9387dd8bd492080b51b45b504d996 P b648efed5499439da654b4c5625fbb5c: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:24.942289 5487 log.cc:824] T 1bb9387dd8bd492080b51b45b504d996 P ff02e3ea75884f589eca57248bd136ad: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:24.942414 5488 tablet_bootstrap.cc:492] T 1bb9387dd8bd492080b51b45b504d996 P 8eed6745a89e450999dcc00e4c0454ef: Bootstrap starting.
I20260619 16:49:24.942456 5489 log.cc:824] T 1bb9387dd8bd492080b51b45b504d996 P b648efed5499439da654b4c5625fbb5c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:24.943202 5488 tablet_bootstrap.cc:654] T 1bb9387dd8bd492080b51b45b504d996 P 8eed6745a89e450999dcc00e4c0454ef: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:24.943423 5487 tablet_bootstrap.cc:492] T 1bb9387dd8bd492080b51b45b504d996 P ff02e3ea75884f589eca57248bd136ad: No bootstrap required, opened a new log
I20260619 16:49:24.943495 5489 tablet_bootstrap.cc:492] T 1bb9387dd8bd492080b51b45b504d996 P b648efed5499439da654b4c5625fbb5c: No bootstrap required, opened a new log
I20260619 16:49:24.943511 5487 ts_tablet_manager.cc:1397] T 1bb9387dd8bd492080b51b45b504d996 P ff02e3ea75884f589eca57248bd136ad: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 16:49:24.943570 5489 ts_tablet_manager.cc:1397] T 1bb9387dd8bd492080b51b45b504d996 P b648efed5499439da654b4c5625fbb5c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 16:49:24.943579 5488 log.cc:824] T 1bb9387dd8bd492080b51b45b504d996 P 8eed6745a89e450999dcc00e4c0454ef: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:24.945906 5487 raft_consensus.cc:348] T 1bb9387dd8bd492080b51b45b504d996 P ff02e3ea75884f589eca57248bd136ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b648efed5499439da654b4c5625fbb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "ff02e3ea75884f589eca57248bd136ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46773 } } peers { permanent_uuid: "8eed6745a89e450999dcc00e4c0454ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260619 16:49:24.946100 5487 raft_consensus.cc:374] T 1bb9387dd8bd492080b51b45b504d996 P ff02e3ea75884f589eca57248bd136ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:24.945989 5489 raft_consensus.cc:348] T 1bb9387dd8bd492080b51b45b504d996 P b648efed5499439da654b4c5625fbb5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b648efed5499439da654b4c5625fbb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "ff02e3ea75884f589eca57248bd136ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46773 } } peers { permanent_uuid: "8eed6745a89e450999dcc00e4c0454ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260619 16:49:24.946125 5487 raft_consensus.cc:729] T 1bb9387dd8bd492080b51b45b504d996 P ff02e3ea75884f589eca57248bd136ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff02e3ea75884f589eca57248bd136ad, State: Initialized, Role: FOLLOWER
I20260619 16:49:24.946132 5489 raft_consensus.cc:374] T 1bb9387dd8bd492080b51b45b504d996 P b648efed5499439da654b4c5625fbb5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:24.946164 5489 raft_consensus.cc:729] T 1bb9387dd8bd492080b51b45b504d996 P b648efed5499439da654b4c5625fbb5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b648efed5499439da654b4c5625fbb5c, State: Initialized, Role: FOLLOWER
I20260619 16:49:24.946328 5489 consensus_queue.cc:260] T 1bb9387dd8bd492080b51b45b504d996 P b648efed5499439da654b4c5625fbb5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b648efed5499439da654b4c5625fbb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "ff02e3ea75884f589eca57248bd136ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46773 } } peers { permanent_uuid: "8eed6745a89e450999dcc00e4c0454ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260619 16:49:24.946341 5487 consensus_queue.cc:260] T 1bb9387dd8bd492080b51b45b504d996 P ff02e3ea75884f589eca57248bd136ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b648efed5499439da654b4c5625fbb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "ff02e3ea75884f589eca57248bd136ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46773 } } peers { permanent_uuid: "8eed6745a89e450999dcc00e4c0454ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260619 16:49:24.946671 5489 ts_tablet_manager.cc:1428] T 1bb9387dd8bd492080b51b45b504d996 P b648efed5499439da654b4c5625fbb5c: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260619 16:49:24.946671 5487 ts_tablet_manager.cc:1428] T 1bb9387dd8bd492080b51b45b504d996 P ff02e3ea75884f589eca57248bd136ad: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260619 16:49:24.946831 5473 heartbeater.cc:503] Master 127.0.98.62:34509 was elected leader, sending a full tablet report...
I20260619 16:49:24.947028 5488 tablet_bootstrap.cc:492] T 1bb9387dd8bd492080b51b45b504d996 P 8eed6745a89e450999dcc00e4c0454ef: No bootstrap required, opened a new log
I20260619 16:49:24.947096 5488 ts_tablet_manager.cc:1397] T 1bb9387dd8bd492080b51b45b504d996 P 8eed6745a89e450999dcc00e4c0454ef: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 16:49:24.949326 5488 raft_consensus.cc:348] T 1bb9387dd8bd492080b51b45b504d996 P 8eed6745a89e450999dcc00e4c0454ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b648efed5499439da654b4c5625fbb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "ff02e3ea75884f589eca57248bd136ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46773 } } peers { permanent_uuid: "8eed6745a89e450999dcc00e4c0454ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260619 16:49:24.949486 5488 raft_consensus.cc:374] T 1bb9387dd8bd492080b51b45b504d996 P 8eed6745a89e450999dcc00e4c0454ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:24.949506 5488 raft_consensus.cc:729] T 1bb9387dd8bd492080b51b45b504d996 P 8eed6745a89e450999dcc00e4c0454ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eed6745a89e450999dcc00e4c0454ef, State: Initialized, Role: FOLLOWER
I20260619 16:49:24.949621 5488 consensus_queue.cc:260] T 1bb9387dd8bd492080b51b45b504d996 P 8eed6745a89e450999dcc00e4c0454ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b648efed5499439da654b4c5625fbb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "ff02e3ea75884f589eca57248bd136ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46773 } } peers { permanent_uuid: "8eed6745a89e450999dcc00e4c0454ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260619 16:49:24.949950 5488 ts_tablet_manager.cc:1428] T 1bb9387dd8bd492080b51b45b504d996 P 8eed6745a89e450999dcc00e4c0454ef: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 16:49:24.949993 5211 heartbeater.cc:503] Master 127.0.98.62:34509 was elected leader, sending a full tablet report...
I20260619 16:49:24.950223 5342 heartbeater.cc:503] Master 127.0.98.62:34509 was elected leader, sending a full tablet report...
W20260619 16:49:25.019253 5343 tablet.cc:2369] T 1bb9387dd8bd492080b51b45b504d996 P b648efed5499439da654b4c5625fbb5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:49:25.124064 5212 tablet.cc:2369] T 1bb9387dd8bd492080b51b45b504d996 P 8eed6745a89e450999dcc00e4c0454ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:49:25.166397 5474 tablet.cc:2369] T 1bb9387dd8bd492080b51b45b504d996 P ff02e3ea75884f589eca57248bd136ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:49:25.337956 5494 raft_consensus.cc:482] T 1bb9387dd8bd492080b51b45b504d996 P b648efed5499439da654b4c5625fbb5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:25.338141 5494 raft_consensus.cc:504] T 1bb9387dd8bd492080b51b45b504d996 P b648efed5499439da654b4c5625fbb5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b648efed5499439da654b4c5625fbb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "ff02e3ea75884f589eca57248bd136ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46773 } } peers { permanent_uuid: "8eed6745a89e450999dcc00e4c0454ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260619 16:49:25.338647 5494 leader_election.cc:288] T 1bb9387dd8bd492080b51b45b504d996 P b648efed5499439da654b4c5625fbb5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff02e3ea75884f589eca57248bd136ad (127.0.98.3:46773), 8eed6745a89e450999dcc00e4c0454ef (127.0.98.1:39659)
I20260619 16:49:25.340277 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:25.339062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48907 (local address 127.0.98.1:39659)
0619 16:49:25.339122 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:25.339124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:25.339134 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:49:25.339187 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:25.339188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:25.339195 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:25.339261 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:25.340032 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:25.340034 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:25.340034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:25.340039 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:25.340061 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:25.340067 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:25.340127 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:25.340179 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 16:49:25.340193 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 16:49:25.340277 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:25.338973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34249 (local address 127.0.98.3:46773)
0619 16:49:25.339030 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:25.339033 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:25.339127 (+ 94us) server_negotiation.cc:407] Connection header received
0619 16:49:25.339187 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:25.339188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:25.339195 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:25.339261 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:25.340032 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:25.340033 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:25.340033 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:25.340039 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:25.340061 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:25.340066 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:25.340127 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:25.340177 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 16:49:25.340193 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 16:49:25.340323 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:25.338902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46773 (local address 127.0.98.2:34249)
0619 16:49:25.339055 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:25.339069 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:25.339120 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:25.339275 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:25.339275 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:25.339279 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:25.339903 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:49:25.339903 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:25.339908 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:49:25.339927 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:25.340124 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:25.340125 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:25.340125 (+ 0us) client_negotiation.cc:769] Sending connection context
0619 16:49:25.340173 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 16:49:25.340205 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":15,"thread_start_us":76,"threads_started":1}
I20260619 16:49:25.340477 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:25.338898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39659 (local address 127.0.98.2:48907)
0619 16:49:25.339028 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:25.339042 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:25.339119 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:25.339275 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:25.339275 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:25.339279 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:25.339886 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:49:25.339886 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:25.339909 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:49:25.339934 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:25.340122 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:25.340124 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:25.340125 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 16:49:25.340172 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 16:49:25.340199 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":96,"threads_started":1}
I20260619 16:49:25.340759 5427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bb9387dd8bd492080b51b45b504d996" candidate_uuid: "b648efed5499439da654b4c5625fbb5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff02e3ea75884f589eca57248bd136ad" is_pre_election: true
I20260619 16:49:25.340807 5165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bb9387dd8bd492080b51b45b504d996" candidate_uuid: "b648efed5499439da654b4c5625fbb5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eed6745a89e450999dcc00e4c0454ef" is_pre_election: true
I20260619 16:49:25.340965 5165 raft_consensus.cc:2454] T 1bb9387dd8bd492080b51b45b504d996 P 8eed6745a89e450999dcc00e4c0454ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b648efed5499439da654b4c5625fbb5c in term 0.
I20260619 16:49:25.340965 5427 raft_consensus.cc:2454] T 1bb9387dd8bd492080b51b45b504d996 P ff02e3ea75884f589eca57248bd136ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b648efed5499439da654b4c5625fbb5c in term 0.
I20260619 16:49:25.341281 5229 leader_election.cc:302] T 1bb9387dd8bd492080b51b45b504d996 P b648efed5499439da654b4c5625fbb5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b648efed5499439da654b4c5625fbb5c, ff02e3ea75884f589eca57248bd136ad; no voters:
I20260619 16:49:25.341567 5494 raft_consensus.cc:2781] T 1bb9387dd8bd492080b51b45b504d996 P b648efed5499439da654b4c5625fbb5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:49:25.341632 5494 raft_consensus.cc:482] T 1bb9387dd8bd492080b51b45b504d996 P b648efed5499439da654b4c5625fbb5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:49:25.341653 5494 raft_consensus.cc:3037] T 1bb9387dd8bd492080b51b45b504d996 P b648efed5499439da654b4c5625fbb5c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:25.342615 5494 raft_consensus.cc:504] T 1bb9387dd8bd492080b51b45b504d996 P b648efed5499439da654b4c5625fbb5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b648efed5499439da654b4c5625fbb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "ff02e3ea75884f589eca57248bd136ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46773 } } peers { permanent_uuid: "8eed6745a89e450999dcc00e4c0454ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260619 16:49:25.342769 5494 leader_election.cc:288] T 1bb9387dd8bd492080b51b45b504d996 P b648efed5499439da654b4c5625fbb5c [CANDIDATE]: Term 1 election: Requested vote from peers ff02e3ea75884f589eca57248bd136ad (127.0.98.3:46773), 8eed6745a89e450999dcc00e4c0454ef (127.0.98.1:39659)
I20260619 16:49:25.342980 5427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bb9387dd8bd492080b51b45b504d996" candidate_uuid: "b648efed5499439da654b4c5625fbb5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff02e3ea75884f589eca57248bd136ad"
I20260619 16:49:25.342988 5165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bb9387dd8bd492080b51b45b504d996" candidate_uuid: "b648efed5499439da654b4c5625fbb5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eed6745a89e450999dcc00e4c0454ef"
I20260619 16:49:25.343062 5427 raft_consensus.cc:3037] T 1bb9387dd8bd492080b51b45b504d996 P ff02e3ea75884f589eca57248bd136ad [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:25.343080 5165 raft_consensus.cc:3037] T 1bb9387dd8bd492080b51b45b504d996 P 8eed6745a89e450999dcc00e4c0454ef [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:25.343866 5427 raft_consensus.cc:2454] T 1bb9387dd8bd492080b51b45b504d996 P ff02e3ea75884f589eca57248bd136ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b648efed5499439da654b4c5625fbb5c in term 1.
I20260619 16:49:25.343981 5165 raft_consensus.cc:2454] T 1bb9387dd8bd492080b51b45b504d996 P 8eed6745a89e450999dcc00e4c0454ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b648efed5499439da654b4c5625fbb5c in term 1.
I20260619 16:49:25.344034 5229 leader_election.cc:302] T 1bb9387dd8bd492080b51b45b504d996 P b648efed5499439da654b4c5625fbb5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b648efed5499439da654b4c5625fbb5c, ff02e3ea75884f589eca57248bd136ad; no voters:
I20260619 16:49:25.344125 5494 raft_consensus.cc:2781] T 1bb9387dd8bd492080b51b45b504d996 P b648efed5499439da654b4c5625fbb5c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:25.344295 5494 raft_consensus.cc:686] T 1bb9387dd8bd492080b51b45b504d996 P b648efed5499439da654b4c5625fbb5c [term 1 LEADER]: Becoming Leader. State: Replica: b648efed5499439da654b4c5625fbb5c, State: Running, Role: LEADER
I20260619 16:49:25.344412 5494 consensus_queue.cc:237] T 1bb9387dd8bd492080b51b45b504d996 P b648efed5499439da654b4c5625fbb5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b648efed5499439da654b4c5625fbb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "ff02e3ea75884f589eca57248bd136ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46773 } } peers { permanent_uuid: "8eed6745a89e450999dcc00e4c0454ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260619 16:49:25.345438 5023 catalog_manager.cc:5495] T 1bb9387dd8bd492080b51b45b504d996 P b648efed5499439da654b4c5625fbb5c reported cstate change: term changed from 0 to 1, leader changed from <none> to b648efed5499439da654b4c5625fbb5c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b648efed5499439da654b4c5625fbb5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b648efed5499439da654b4c5625fbb5c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff02e3ea75884f589eca57248bd136ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8eed6745a89e450999dcc00e4c0454ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } health_report { overall_health: UNKNOWN } } }
I20260619 16:49:25.382130 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:25.380757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49416 (local address 127.0.98.2:40739)
0619 16:49:25.380809 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:25.380810 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:25.380822 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:49:25.380919 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:25.380920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:25.380940 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:25.380997 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:25.381812 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:25.381814 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:25.381814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:25.381819 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:25.381842 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:25.381847 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:25.381906 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:25.382006 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 16:49:25.382020 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 16:49:25.384414 5427 raft_consensus.cc:1261] T 1bb9387dd8bd492080b51b45b504d996 P ff02e3ea75884f589eca57248bd136ad [term 1 FOLLOWER]: Refusing update from remote peer b648efed5499439da654b4c5625fbb5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:25.384414 5165 raft_consensus.cc:1261] T 1bb9387dd8bd492080b51b45b504d996 P 8eed6745a89e450999dcc00e4c0454ef [term 1 FOLLOWER]: Refusing update from remote peer b648efed5499439da654b4c5625fbb5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:25.384856 5498 consensus_queue.cc:1035] T 1bb9387dd8bd492080b51b45b504d996 P b648efed5499439da654b4c5625fbb5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff02e3ea75884f589eca57248bd136ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:25.385020 5494 consensus_queue.cc:1035] T 1bb9387dd8bd492080b51b45b504d996 P b648efed5499439da654b4c5625fbb5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eed6745a89e450999dcc00e4c0454ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:25.387791 5501 mvcc.cc:205] Tried to move back new op lower bound from 7298612287011377152 to 7298612286851317760. Current Snapshot: MvccSnapshot[applied={T|T < 7298612287011377152}]
I20260619 16:49:25.387856 5502 mvcc.cc:205] Tried to move back new op lower bound from 7298612287011377152 to 7298612286851317760. Current Snapshot: MvccSnapshot[applied={T|T < 7298612287011377152}]
I20260619 16:49:25.389743 5504 mvcc.cc:205] Tried to move back new op lower bound from 7298612287011377152 to 7298612286851317760. Current Snapshot: MvccSnapshot[applied={T|T < 7298612287011377152}]
I20260619 16:49:25.391065 5022 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:25.391180 5022 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:25.392102 5022 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7d7778a4c4ad40cd96e424008990d34e: Sending DeleteTablet for 3 replicas of tablet 1bb9387dd8bd492080b51b45b504d996
I20260619 16:49:25.392480 5276 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bb9387dd8bd492080b51b45b504d996 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:25 UTC) from {username='slave'} at 127.0.0.1:49406
I20260619 16:49:25.392653 5407 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bb9387dd8bd492080b51b45b504d996 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:25 UTC) from {username='slave'} at 127.0.0.1:49148
I20260619 16:49:25.392786 5145 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bb9387dd8bd492080b51b45b504d996 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:25 UTC) from {username='slave'} at 127.0.0.1:32934
I20260619 16:49:25.393062 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 5083
I20260619 16:49:25.394101 5510 tablet_replica.cc:330] T 1bb9387dd8bd492080b51b45b504d996 P b648efed5499439da654b4c5625fbb5c: stopping tablet replica
I20260619 16:49:25.394271 5510 raft_consensus.cc:2229] T 1bb9387dd8bd492080b51b45b504d996 P b648efed5499439da654b4c5625fbb5c [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:49:25.394347 5511 tablet_replica.cc:330] T 1bb9387dd8bd492080b51b45b504d996 P ff02e3ea75884f589eca57248bd136ad: stopping tablet replica
I20260619 16:49:25.394400 5510 raft_consensus.cc:2258] T 1bb9387dd8bd492080b51b45b504d996 P b648efed5499439da654b4c5625fbb5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:25.394621 5511 raft_consensus.cc:2229] T 1bb9387dd8bd492080b51b45b504d996 P ff02e3ea75884f589eca57248bd136ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:25.394843 5511 raft_consensus.cc:2258] T 1bb9387dd8bd492080b51b45b504d996 P ff02e3ea75884f589eca57248bd136ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:25.395282 5510 ts_tablet_manager.cc:1905] T 1bb9387dd8bd492080b51b45b504d996 P b648efed5499439da654b4c5625fbb5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:25.396011 5511 ts_tablet_manager.cc:1905] T 1bb9387dd8bd492080b51b45b504d996 P ff02e3ea75884f589eca57248bd136ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:25.397581 5510 ts_tablet_manager.cc:1918] T 1bb9387dd8bd492080b51b45b504d996 P b648efed5499439da654b4c5625fbb5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:25.397665 5510 log.cc:1196] T 1bb9387dd8bd492080b51b45b504d996 P b648efed5499439da654b4c5625fbb5c: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-1/wal/wals/1bb9387dd8bd492080b51b45b504d996
I20260619 16:49:25.397878 5511 ts_tablet_manager.cc:1918] T 1bb9387dd8bd492080b51b45b504d996 P ff02e3ea75884f589eca57248bd136ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:25.397948 5510 ts_tablet_manager.cc:1939] T 1bb9387dd8bd492080b51b45b504d996 P b648efed5499439da654b4c5625fbb5c: Deleting consensus metadata
I20260619 16:49:25.397964 5511 log.cc:1196] T 1bb9387dd8bd492080b51b45b504d996 P ff02e3ea75884f589eca57248bd136ad: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781887741903642-392-0/minicluster-data/ts-2/wal/wals/1bb9387dd8bd492080b51b45b504d996
I20260619 16:49:25.398270 5511 ts_tablet_manager.cc:1939] T 1bb9387dd8bd492080b51b45b504d996 P ff02e3ea75884f589eca57248bd136ad: Deleting consensus metadata
I20260619 16:49:25.398535 5008 catalog_manager.cc:4841] TS b648efed5499439da654b4c5625fbb5c (127.0.98.2:40739): tablet 1bb9387dd8bd492080b51b45b504d996 (table test-table [id=05361f2aee664118bb1b5b30561c38c1]) successfully deleted
I20260619 16:49:25.398900 5007 catalog_manager.cc:4841] TS ff02e3ea75884f589eca57248bd136ad (127.0.98.3:46773): tablet 1bb9387dd8bd492080b51b45b504d996 (table test-table [id=05361f2aee664118bb1b5b30561c38c1]) successfully deleted
W20260619 16:49:25.401651 5010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39659 (error 108)
I20260619 16:49:25.401865 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 5214
I20260619 16:49:25.402892 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:25.402711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39659 (local address 127.0.0.1:32938)
0619 16:49:25.402783 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:25.402846 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 16:49:25.403467 5010 catalog_manager.cc:4584] TS 8eed6745a89e450999dcc00e4c0454ef (127.0.98.1:39659): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1bb9387dd8bd492080b51b45b504d996: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39659: connect: Connection refused (error 111)
I20260619 16:49:25.407153 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 5345
I20260619 16:49:25.413034 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 4992
2026-06-19T16:49:25Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1127 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 16:49:25.433869 392 mini_oidc.cc:152] Starting JWKS server
I20260619 16:49:25.435237 392 webserver.cc:460] Webserver started at https://127.0.0.1:44987/ using document root <none> and password file <none>
I20260619 16:49:25.435350 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 16:49:25.435534 392 webserver.cc:460] Webserver started at http://127.0.0.1:34107/ using document root <none> and password file <none>
2026-06-19T16:49:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:49:25Z Disabled control of system clock
I20260619 16:49:25.446048 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45027
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45027
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44987/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887741903642-392-0/testchainca.cert with env {}
W20260619 16:49:25.550688 5519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:25.551008 5519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:25.551039 5519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:25.552640 5519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:25.552711 5519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:25.552723 5519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:25.552736 5519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:25.552748 5519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 16:49:25.552760 5519 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 16:49:25.552770 5519 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44987/jwks.json
I20260619 16:49:25.554864 5519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45027
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44987/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887741903642-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:25.555174 5519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:25.555413 5519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:25.561602 5525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:25.561602 5524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:25.561602 5527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:25.562355 5519 server_base.cc:1034] running on GCE node
I20260619 16:49:25.562629 5519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:25.562958 5519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:25.564134 5519 hybrid_clock.cc:648] HybridClock initialized: now 1781887765564098 us; error 52 us; skew 500 ppm
I20260619 16:49:25.565733 5519 webserver.cc:460] Webserver started at http://127.0.98.62:41531/ using document root <none> and password file <none>
I20260619 16:49:25.565981 5519 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:25.566030 5519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:25.566140 5519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:25.567402 5519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "636e8d43f6e14f4f84b112bd9e85fd7b"
format_stamp: "Formatted at 2026-06-19 16:49:25 on dist-test-slave-jsw9"
server_key: "2faf4cde07ebfded9024097828b7560f"
server_key_iv: "ff1f7544619e5fb23509a074a402c3b1"
server_key_version: "encryptionkey@0"
I20260619 16:49:25.567740 5519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "636e8d43f6e14f4f84b112bd9e85fd7b"
format_stamp: "Formatted at 2026-06-19 16:49:25 on dist-test-slave-jsw9"
server_key: "2faf4cde07ebfded9024097828b7560f"
server_key_iv: "ff1f7544619e5fb23509a074a402c3b1"
server_key_version: "encryptionkey@0"
I20260619 16:49:25.569128 5519 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 16:49:25.569955 5533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:25.570113 5519 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:25.570227 5519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "636e8d43f6e14f4f84b112bd9e85fd7b"
format_stamp: "Formatted at 2026-06-19 16:49:25 on dist-test-slave-jsw9"
server_key: "2faf4cde07ebfded9024097828b7560f"
server_key_iv: "ff1f7544619e5fb23509a074a402c3b1"
server_key_version: "encryptionkey@0"
I20260619 16:49:25.570302 5519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:25.591145 5519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:25.591691 5519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:25.591902 5519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:25.597040 5519 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45027
I20260619 16:49:25.597263 5585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45027 every 8 connection(s)
I20260619 16:49:25.597918 5519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:25.599758 5586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:25.602891 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 5519
I20260619 16:49:25.603147 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:49:25.603600 392 external_mini_cluster.cc:1428] Setting key 058566f42dc1d7c7ba0e2352029d7c25
I20260619 16:49:25.603951 5586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 636e8d43f6e14f4f84b112bd9e85fd7b: Bootstrap starting.
I20260619 16:49:25.605194 5586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 636e8d43f6e14f4f84b112bd9e85fd7b: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:25.605615 5586 log.cc:824] T 00000000000000000000000000000000 P 636e8d43f6e14f4f84b112bd9e85fd7b: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:25.606698 5586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 636e8d43f6e14f4f84b112bd9e85fd7b: No bootstrap required, opened a new log
I20260619 16:49:25.609227 5586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 636e8d43f6e14f4f84b112bd9e85fd7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "636e8d43f6e14f4f84b112bd9e85fd7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45027 } }
I20260619 16:49:25.609460 5586 raft_consensus.cc:374] T 00000000000000000000000000000000 P 636e8d43f6e14f4f84b112bd9e85fd7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:25.609503 5586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 636e8d43f6e14f4f84b112bd9e85fd7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 636e8d43f6e14f4f84b112bd9e85fd7b, State: Initialized, Role: FOLLOWER
I20260619 16:49:25.609664 5586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 636e8d43f6e14f4f84b112bd9e85fd7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "636e8d43f6e14f4f84b112bd9e85fd7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45027 } }
I20260619 16:49:25.609752 5586 raft_consensus.cc:388] T 00000000000000000000000000000000 P 636e8d43f6e14f4f84b112bd9e85fd7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:25.609787 5586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 636e8d43f6e14f4f84b112bd9e85fd7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:25.609838 5586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 636e8d43f6e14f4f84b112bd9e85fd7b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:25.610857 5586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 636e8d43f6e14f4f84b112bd9e85fd7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "636e8d43f6e14f4f84b112bd9e85fd7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45027 } }
I20260619 16:49:25.610992 5586 leader_election.cc:302] T 00000000000000000000000000000000 P 636e8d43f6e14f4f84b112bd9e85fd7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 636e8d43f6e14f4f84b112bd9e85fd7b; no voters:
I20260619 16:49:25.611228 5586 leader_election.cc:288] T 00000000000000000000000000000000 P 636e8d43f6e14f4f84b112bd9e85fd7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:49:25.611346 5591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 636e8d43f6e14f4f84b112bd9e85fd7b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:25.611347 5589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:25.604555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42018 (local address 127.0.98.62:45027)
0619 16:49:25.604854 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:25.604865 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:25.604905 (+ 40us) server_negotiation.cc:407] Connection header received
0619 16:49:25.605668 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:25.605680 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:25.605783 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:25.605972 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:25.606323 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:25.607802 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:25.608742 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:25.609416 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:25.609552 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:25.610436 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:25.610442 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:25.610442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:25.610712 (+ 270us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:25.610767 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:25.610787 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:25.610857 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:25.610985 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 16:49:25.611088 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":121,"threads_started":1}
I20260619 16:49:25.611529 5591 raft_consensus.cc:686] T 00000000000000000000000000000000 P 636e8d43f6e14f4f84b112bd9e85fd7b [term 1 LEADER]: Becoming Leader. State: Replica: 636e8d43f6e14f4f84b112bd9e85fd7b, State: Running, Role: LEADER
I20260619 16:49:25.611640 5591 consensus_queue.cc:237] T 00000000000000000000000000000000 P 636e8d43f6e14f4f84b112bd9e85fd7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "636e8d43f6e14f4f84b112bd9e85fd7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45027 } }
I20260619 16:49:25.611626 5586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 636e8d43f6e14f4f84b112bd9e85fd7b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:25.612339 5592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 636e8d43f6e14f4f84b112bd9e85fd7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 636e8d43f6e14f4f84b112bd9e85fd7b. Latest consensus state: current_term: 1 leader_uuid: "636e8d43f6e14f4f84b112bd9e85fd7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "636e8d43f6e14f4f84b112bd9e85fd7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45027 } } }
I20260619 16:49:25.612339 5593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 636e8d43f6e14f4f84b112bd9e85fd7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "636e8d43f6e14f4f84b112bd9e85fd7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "636e8d43f6e14f4f84b112bd9e85fd7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45027 } } }
I20260619 16:49:25.612448 5592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 636e8d43f6e14f4f84b112bd9e85fd7b [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:25.612464 5593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 636e8d43f6e14f4f84b112bd9e85fd7b [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:25.612658 5595 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:25.612972 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:49:25.613262 5595 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:25.615286 5595 catalog_manager.cc:1347] Generated new cluster ID: 1fb526acd93945e4a2946111ce6f2464
I20260619 16:49:25.615356 5595 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:49:25.629499 5595 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:49:25.630133 5595 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:25.638792 5595 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 636e8d43f6e14f4f84b112bd9e85fd7b: Generated new TSK 0
I20260619 16:49:25.639026 5595 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 16:49:25.665642 5616 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 16:49:25.665786 5616 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 16:49:25.666051 5589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:25.646534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42030 (local address 127.0.98.62:45027)
0619 16:49:25.646594 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:25.646596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:25.646778 (+ 182us) server_negotiation.cc:407] Connection header received
0619 16:49:25.646895 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:25.646896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:25.646911 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:25.646982 (+ 71us) server_negotiation.cc:226] Negotiated authn=JWT
0619 16:49:25.647163 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:25.647721 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:25.648977 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:25.649297 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:25.649390 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:25.649732 (+ 342us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 16:49:25.665771 (+ 16039us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44987/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 16:49:25.665973 (+ 202us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42030: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44987/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":31}
W20260619 16:49:25.666229 5589 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42030: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44987/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 16:49:25.667006 5615 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:25.646447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45027 (local address 127.0.0.1:42030)
0619 16:49:25.646670 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:25.646684 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:25.646775 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:25.647003 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:25.647004 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:25.647008 (+ 4us) client_negotiation.cc:189] Negotiated authn=JWT
0619 16:49:25.647102 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:25.647105 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:25.647794 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:25.647795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:25.648836 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:25.648841 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:25.649412 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:25.649418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:25.649514 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:25.649558 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 16:49:25.666111 (+ 16553us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44987/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 16:49:25.666329 (+ 218us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:45027: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44987/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":96,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:45027: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44987/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 16:49:25.667667 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 5519
2026-06-19T16:49:25Z chronyd exiting
I20260619 16:49:25.705440 392 test_util.cc:175] -----------------------------------------------
I20260619 16:49:25.705565 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781887741903642-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (273 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 16:49:25.706737 392 mini_oidc.cc:152] Starting JWKS server
I20260619 16:49:25.707520 392 webserver.cc:460] Webserver started at https://127.0.0.1:42221/ using document root <none> and password file <none>
I20260619 16:49:25.707571 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 16:49:25.707717 392 webserver.cc:460] Webserver started at http://127.0.0.1:42057/ using document root <none> and password file <none>
2026-06-19T16:49:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:49:25Z Disabled control of system clock
I20260619 16:49:25.718741 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34059
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34059
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42221/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887741903642-392-0/testchainca.cert with env {}
W20260619 16:49:25.823309 5622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:25.823634 5622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:25.823662 5622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:25.825330 5622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:25.825397 5622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:25.825409 5622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:25.825423 5622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:25.825434 5622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 16:49:25.825446 5622 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 16:49:25.825455 5622 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42221/jwks.json
I20260619 16:49:25.827548 5622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34059
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42221/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887741903642-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:25.827911 5622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:25.828163 5622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:25.833716 5630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:25.833727 5628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:25.833712 5627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:25.834082 5622 server_base.cc:1034] running on GCE node
I20260619 16:49:25.834385 5622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:25.834724 5622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:25.835915 5622 hybrid_clock.cc:648] HybridClock initialized: now 1781887765835866 us; error 61 us; skew 500 ppm
I20260619 16:49:25.837469 5622 webserver.cc:460] Webserver started at http://127.0.98.62:34009/ using document root <none> and password file <none>
I20260619 16:49:25.837774 5622 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:25.837833 5622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:25.837971 5622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:25.839283 5622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "b1b7c8c833be4223a4a579c027011edf"
format_stamp: "Formatted at 2026-06-19 16:49:25 on dist-test-slave-jsw9"
server_key: "e73e1e1aeb4ca4cd9307244177069a0d"
server_key_iv: "a5faa912d596f980b6b43a66ba91efd7"
server_key_version: "encryptionkey@0"
I20260619 16:49:25.839684 5622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "b1b7c8c833be4223a4a579c027011edf"
format_stamp: "Formatted at 2026-06-19 16:49:25 on dist-test-slave-jsw9"
server_key: "e73e1e1aeb4ca4cd9307244177069a0d"
server_key_iv: "a5faa912d596f980b6b43a66ba91efd7"
server_key_version: "encryptionkey@0"
I20260619 16:49:25.841135 5622 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 16:49:25.841995 5636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:25.842211 5622 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:25.842309 5622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "b1b7c8c833be4223a4a579c027011edf"
format_stamp: "Formatted at 2026-06-19 16:49:25 on dist-test-slave-jsw9"
server_key: "e73e1e1aeb4ca4cd9307244177069a0d"
server_key_iv: "a5faa912d596f980b6b43a66ba91efd7"
server_key_version: "encryptionkey@0"
I20260619 16:49:25.842386 5622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:25.856210 5622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:25.856613 5622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:25.856769 5622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:25.861622 5622 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34059
I20260619 16:49:25.861702 5688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34059 every 8 connection(s)
I20260619 16:49:25.862102 5622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:25.863296 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:25.865168 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 5622
I20260619 16:49:25.865278 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:49:25.865494 392 external_mini_cluster.cc:1428] Setting key cd143430c1668ee7b92d0e6b5d2cb027
I20260619 16:49:25.866660 5689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1b7c8c833be4223a4a579c027011edf: Bootstrap starting.
I20260619 16:49:25.867650 5689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1b7c8c833be4223a4a579c027011edf: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:25.868046 5689 log.cc:824] T 00000000000000000000000000000000 P b1b7c8c833be4223a4a579c027011edf: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:25.869016 5689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1b7c8c833be4223a4a579c027011edf: No bootstrap required, opened a new log
I20260619 16:49:25.871524 5689 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1b7c8c833be4223a4a579c027011edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b7c8c833be4223a4a579c027011edf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34059 } }
I20260619 16:49:25.871716 5689 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1b7c8c833be4223a4a579c027011edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:25.871752 5689 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1b7c8c833be4223a4a579c027011edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1b7c8c833be4223a4a579c027011edf, State: Initialized, Role: FOLLOWER
I20260619 16:49:25.871929 5689 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1b7c8c833be4223a4a579c027011edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b7c8c833be4223a4a579c027011edf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34059 } }
I20260619 16:49:25.872030 5689 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1b7c8c833be4223a4a579c027011edf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:25.872063 5689 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1b7c8c833be4223a4a579c027011edf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:25.872115 5689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1b7c8c833be4223a4a579c027011edf [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:25.872712 5692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:25.866327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34238 (local address 127.0.98.62:34059)
0619 16:49:25.866594 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:25.866606 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:25.866642 (+ 36us) server_negotiation.cc:407] Connection header received
0619 16:49:25.867327 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:25.867337 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:25.867432 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:25.867699 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:25.867970 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:25.869328 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:25.870244 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:25.870969 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:25.871133 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:25.871892 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:25.871898 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:25.871898 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:25.872151 (+ 253us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:25.872201 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:25.872221 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:25.872300 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:25.872361 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 16:49:25.872466 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":98,"threads_started":1}
I20260619 16:49:25.873037 5689 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1b7c8c833be4223a4a579c027011edf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b7c8c833be4223a4a579c027011edf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34059 } }
I20260619 16:49:25.873153 5689 leader_election.cc:302] T 00000000000000000000000000000000 P b1b7c8c833be4223a4a579c027011edf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1b7c8c833be4223a4a579c027011edf; no voters:
I20260619 16:49:25.873355 5689 leader_election.cc:288] T 00000000000000000000000000000000 P b1b7c8c833be4223a4a579c027011edf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:49:25.873416 5694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1b7c8c833be4223a4a579c027011edf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:25.873579 5694 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1b7c8c833be4223a4a579c027011edf [term 1 LEADER]: Becoming Leader. State: Replica: b1b7c8c833be4223a4a579c027011edf, State: Running, Role: LEADER
I20260619 16:49:25.873660 5689 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1b7c8c833be4223a4a579c027011edf [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:25.873683 5694 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1b7c8c833be4223a4a579c027011edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b7c8c833be4223a4a579c027011edf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34059 } }
I20260619 16:49:25.873886 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:49:25.874752 5696 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1b7c8c833be4223a4a579c027011edf [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1b7c8c833be4223a4a579c027011edf. Latest consensus state: current_term: 1 leader_uuid: "b1b7c8c833be4223a4a579c027011edf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b7c8c833be4223a4a579c027011edf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34059 } } }
I20260619 16:49:25.874847 5696 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1b7c8c833be4223a4a579c027011edf [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:25.875053 5695 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1b7c8c833be4223a4a579c027011edf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1b7c8c833be4223a4a579c027011edf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b7c8c833be4223a4a579c027011edf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34059 } } }
I20260619 16:49:25.875139 5695 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1b7c8c833be4223a4a579c027011edf [sys.catalog]: This master's current role is: LEADER
W20260619 16:49:25.876470 5710 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b1b7c8c833be4223a4a579c027011edf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:49:25.876561 5710 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:49:25.876657 5707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:25.876905 5707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:25.878865 5707 catalog_manager.cc:1347] Generated new cluster ID: babb1405957f4ea2ad0e24e80f6b4fe7
I20260619 16:49:25.878937 5707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:49:25.907219 5707 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:49:25.908133 5707 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:25.914803 5707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1b7c8c833be4223a4a579c027011edf: Generated new TSK 0
I20260619 16:49:25.915094 5707 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 16:49:25.947949 5719 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 16:49:25.948050 5719 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 16:49:25.948540 5692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:25.932487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34242 (local address 127.0.98.62:34059)
0619 16:49:25.932545 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:25.932548 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:25.932634 (+ 86us) server_negotiation.cc:407] Connection header received
0619 16:49:25.932708 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:25.932715 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:25.932731 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:25.932830 (+ 99us) server_negotiation.cc:226] Negotiated authn=JWT
0619 16:49:25.932981 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:25.933903 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:25.935188 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:25.935517 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:25.935601 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:25.935857 (+ 256us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 16:49:25.948228 (+ 12371us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42221/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 16:49:25.948448 (+ 220us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34242: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42221/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":31}
W20260619 16:49:25.948608 5692 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34242: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42221/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260619 16:49:25.948668 5718 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:25.932390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34059 (local address 127.0.0.1:34242)
0619 16:49:25.932555 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:25.932567 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:25.932638 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:25.932849 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:25.932850 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:25.932853 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0619 16:49:25.932922 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:25.932925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:25.934128 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:25.934130 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:25.935081 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:25.935087 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:25.935620 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:25.935625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:25.935696 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:25.935703 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 16:49:25.948417 (+ 12714us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42221/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 16:49:25.948552 (+ 135us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34059: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42221/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":96,"threads_started":1}
I20260619 16:49:25.949126 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 5622
2026-06-19T16:49:25Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (281 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 16:49:25.988102 392 mini_oidc.cc:152] Starting JWKS server
I20260619 16:49:25.988900 392 webserver.cc:460] Webserver started at https://127.0.0.1:39089/ using document root <none> and password file <none>
I20260619 16:49:25.988955 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 16:49:25.989107 392 webserver.cc:460] Webserver started at http://127.0.0.1:42437/ using document root <none> and password file <none>
2026-06-19T16:49:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:49:25Z Disabled control of system clock
I20260619 16:49:25.998764 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44693
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44693
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39089/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 16:49:26.102516 5725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:26.102843 5725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:26.102871 5725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:26.104481 5725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:26.104547 5725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:26.104560 5725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:26.104573 5725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:26.104584 5725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 16:49:26.104596 5725 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 16:49:26.104607 5725 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39089/jwks.json
I20260619 16:49:26.106478 5725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44693
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39089/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:26.106810 5725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:26.107156 5725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:49:26.112963 5725 server_base.cc:1034] running on GCE node
W20260619 16:49:26.113122 5731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:26.113042 5733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:26.112984 5730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:26.113581 5725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:26.113941 5725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:26.115139 5725 hybrid_clock.cc:648] HybridClock initialized: now 1781887766115120 us; error 41 us; skew 500 ppm
I20260619 16:49:26.117194 5725 webserver.cc:460] Webserver started at http://127.0.98.62:37441/ using document root <none> and password file <none>
I20260619 16:49:26.117537 5725 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:26.117611 5725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:26.117873 5725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:26.119323 5725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "04082b9eddea4c61a456a950cfa9c156"
format_stamp: "Formatted at 2026-06-19 16:49:26 on dist-test-slave-jsw9"
server_key: "90f6539ca1abdb480ed1b086f9bee256"
server_key_iv: "5f87f17ad0c18cd1721729a65239848b"
server_key_version: "encryptionkey@0"
I20260619 16:49:26.119745 5725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "04082b9eddea4c61a456a950cfa9c156"
format_stamp: "Formatted at 2026-06-19 16:49:26 on dist-test-slave-jsw9"
server_key: "90f6539ca1abdb480ed1b086f9bee256"
server_key_iv: "5f87f17ad0c18cd1721729a65239848b"
server_key_version: "encryptionkey@0"
I20260619 16:49:26.121325 5725 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 16:49:26.122409 5739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:26.122584 5725 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:26.122686 5725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "04082b9eddea4c61a456a950cfa9c156"
format_stamp: "Formatted at 2026-06-19 16:49:26 on dist-test-slave-jsw9"
server_key: "90f6539ca1abdb480ed1b086f9bee256"
server_key_iv: "5f87f17ad0c18cd1721729a65239848b"
server_key_version: "encryptionkey@0"
I20260619 16:49:26.122769 5725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:26.137622 5725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:26.138055 5725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:26.138243 5725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:26.143965 5725 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44693
I20260619 16:49:26.143998 5791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44693 every 8 connection(s)
I20260619 16:49:26.144418 5725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:26.145493 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 5725
I20260619 16:49:26.145612 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:49:26.145577 5792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:26.145839 392 external_mini_cluster.cc:1428] Setting key badc79b68b81f16224fb9aacd394c87c
I20260619 16:49:26.150491 5792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04082b9eddea4c61a456a950cfa9c156: Bootstrap starting.
I20260619 16:49:26.151398 5792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04082b9eddea4c61a456a950cfa9c156: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:26.151767 5792 log.cc:824] T 00000000000000000000000000000000 P 04082b9eddea4c61a456a950cfa9c156: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:26.152777 5792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04082b9eddea4c61a456a950cfa9c156: No bootstrap required, opened a new log
I20260619 16:49:26.153436 5795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:26.146572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49914 (local address 127.0.98.62:44693)
0619 16:49:26.146795 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:26.146807 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:26.146844 (+ 37us) server_negotiation.cc:407] Connection header received
0619 16:49:26.147703 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:26.147717 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:26.147850 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:26.148062 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:26.148326 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:26.149712 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:26.150714 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:26.151373 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:26.151546 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:26.152502 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:26.152508 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:26.152508 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:26.152794 (+ 286us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:26.152853 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:26.152876 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:26.152959 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:26.153065 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 16:49:26.153189 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
I20260619 16:49:26.154635 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:49:26.155231 5792 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04082b9eddea4c61a456a950cfa9c156 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04082b9eddea4c61a456a950cfa9c156" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44693 } }
I20260619 16:49:26.155419 5792 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04082b9eddea4c61a456a950cfa9c156 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:26.155471 5792 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04082b9eddea4c61a456a950cfa9c156 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04082b9eddea4c61a456a950cfa9c156, State: Initialized, Role: FOLLOWER
I20260619 16:49:26.155611 5792 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04082b9eddea4c61a456a950cfa9c156 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04082b9eddea4c61a456a950cfa9c156" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44693 } }
I20260619 16:49:26.155699 5792 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04082b9eddea4c61a456a950cfa9c156 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:26.155742 5792 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04082b9eddea4c61a456a950cfa9c156 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:26.155794 5792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04082b9eddea4c61a456a950cfa9c156 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:26.156500 5792 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04082b9eddea4c61a456a950cfa9c156 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04082b9eddea4c61a456a950cfa9c156" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44693 } }
I20260619 16:49:26.156600 5792 leader_election.cc:302] T 00000000000000000000000000000000 P 04082b9eddea4c61a456a950cfa9c156 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04082b9eddea4c61a456a950cfa9c156; no voters:
I20260619 16:49:26.156769 5792 leader_election.cc:288] T 00000000000000000000000000000000 P 04082b9eddea4c61a456a950cfa9c156 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:49:26.156840 5797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04082b9eddea4c61a456a950cfa9c156 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:26.157001 5797 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04082b9eddea4c61a456a950cfa9c156 [term 1 LEADER]: Becoming Leader. State: Replica: 04082b9eddea4c61a456a950cfa9c156, State: Running, Role: LEADER
I20260619 16:49:26.157066 5792 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04082b9eddea4c61a456a950cfa9c156 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:26.157096 5797 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04082b9eddea4c61a456a950cfa9c156 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04082b9eddea4c61a456a950cfa9c156" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44693 } }
I20260619 16:49:26.157581 5798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04082b9eddea4c61a456a950cfa9c156 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04082b9eddea4c61a456a950cfa9c156" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04082b9eddea4c61a456a950cfa9c156" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44693 } } }
I20260619 16:49:26.157591 5799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04082b9eddea4c61a456a950cfa9c156 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04082b9eddea4c61a456a950cfa9c156. Latest consensus state: current_term: 1 leader_uuid: "04082b9eddea4c61a456a950cfa9c156" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04082b9eddea4c61a456a950cfa9c156" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44693 } } }
I20260619 16:49:26.157742 5798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04082b9eddea4c61a456a950cfa9c156 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:26.157755 5799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04082b9eddea4c61a456a950cfa9c156 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:26.158363 5806 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:26.159191 5806 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:26.161059 5806 catalog_manager.cc:1347] Generated new cluster ID: 99e51f528eec434791f9e3076583a456
I20260619 16:49:26.161131 5806 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:49:26.165506 5806 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:49:26.166337 5806 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:26.175001 5806 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04082b9eddea4c61a456a950cfa9c156: Generated new TSK 0
I20260619 16:49:26.175275 5806 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:49:26.192121 5795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:26.187579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49918 (local address 127.0.98.62:44693)
0619 16:49:26.187646 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:26.187649 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:26.187716 (+ 67us) server_negotiation.cc:407] Connection header received
0619 16:49:26.187799 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:26.187800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:26.187816 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:26.187920 (+ 104us) server_negotiation.cc:226] Negotiated authn=JWT
0619 16:49:26.188050 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:26.188952 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:26.189955 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:26.190318 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:26.190391 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:26.190600 (+ 209us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 16:49:26.191865 (+ 1265us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39089/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 16:49:26.192022 (+ 157us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49918: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39089/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":38}
W20260619 16:49:26.192205 5795 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49918: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39089/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 16:49:26.194152 5821 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:26.187442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44693 (local address 127.0.0.1:49918)
0619 16:49:26.187625 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:26.187637 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:26.187723 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:26.187920 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:26.187921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:26.187924 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0619 16:49:26.187989 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:26.187992 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:26.189071 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:26.189072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:26.189868 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:26.189871 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:26.190400 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:26.190410 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:26.190470 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:26.190478 (+ 8us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 16:49:26.191992 (+ 1514us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39089/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 16:49:26.194037 (+ 2045us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:44693: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39089/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":1789,"thread_start_us":101,"threads_started":1}
I20260619 16:49:26.194815 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 5725
E20260619 16:49:26.203871 5822 webserver.cc:533] Webserver: sslize failed:
E20260619 16:49:26.204068 5822 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-19T16:49:26Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (245 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T16:49:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:49:26Z Disabled control of system clock
I20260619 16:49:26.242408 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43367
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43367
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887741903642-392-0/testchainca.cert with env {}
W20260619 16:49:26.344848 5826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:26.345165 5826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:26.345196 5826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:26.346884 5826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:26.346975 5826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:26.346988 5826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:26.347000 5826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:26.347012 5826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 16:49:26.347025 5826 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 16:49:26.347034 5826 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 16:49:26.349391 5826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43367
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887741903642-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:26.349789 5826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:26.350081 5826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:49:26.356173 5826 server_base.cc:1034] running on GCE node
W20260619 16:49:26.356118 5832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:26.356118 5831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:26.356118 5834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:26.356716 5826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:26.357110 5826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:26.358265 5826 hybrid_clock.cc:648] HybridClock initialized: now 1781887766358242 us; error 41 us; skew 500 ppm
I20260619 16:49:26.359758 5826 webserver.cc:460] Webserver started at http://127.0.98.62:39959/ using document root <none> and password file <none>
I20260619 16:49:26.360033 5826 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:26.360083 5826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:26.360208 5826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:26.361452 5826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "2c41fa5d42ec417aad2dde4fdad99d78"
format_stamp: "Formatted at 2026-06-19 16:49:26 on dist-test-slave-jsw9"
server_key: "ccf5738b8c04e2e50c2d1ed762b92870"
server_key_iv: "c22a17d3d397a5606ca03ceea78771ad"
server_key_version: "encryptionkey@0"
I20260619 16:49:26.361846 5826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "2c41fa5d42ec417aad2dde4fdad99d78"
format_stamp: "Formatted at 2026-06-19 16:49:26 on dist-test-slave-jsw9"
server_key: "ccf5738b8c04e2e50c2d1ed762b92870"
server_key_iv: "c22a17d3d397a5606ca03ceea78771ad"
server_key_version: "encryptionkey@0"
I20260619 16:49:26.363481 5826 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 16:49:26.364426 5840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:26.364638 5826 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:49:26.364737 5826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "2c41fa5d42ec417aad2dde4fdad99d78"
format_stamp: "Formatted at 2026-06-19 16:49:26 on dist-test-slave-jsw9"
server_key: "ccf5738b8c04e2e50c2d1ed762b92870"
server_key_iv: "c22a17d3d397a5606ca03ceea78771ad"
server_key_version: "encryptionkey@0"
I20260619 16:49:26.364847 5826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:26.387180 5826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:26.387735 5826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:26.387908 5826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:26.393419 5826 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43367
I20260619 16:49:26.393492 5892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43367 every 8 connection(s)
I20260619 16:49:26.393913 5826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:26.395160 5893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:26.397930 5893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c41fa5d42ec417aad2dde4fdad99d78: Bootstrap starting.
I20260619 16:49:26.398689 5893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c41fa5d42ec417aad2dde4fdad99d78: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:26.399003 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 5826
I20260619 16:49:26.399071 5893 log.cc:824] T 00000000000000000000000000000000 P 2c41fa5d42ec417aad2dde4fdad99d78: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:26.399113 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:49:26.399345 392 external_mini_cluster.cc:1428] Setting key e6df59a1a62ec8cf260734fd4893025a
I20260619 16:49:26.399986 5893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c41fa5d42ec417aad2dde4fdad99d78: No bootstrap required, opened a new log
I20260619 16:49:26.401904 5893 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c41fa5d42ec417aad2dde4fdad99d78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c41fa5d42ec417aad2dde4fdad99d78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43367 } }
I20260619 16:49:26.402068 5893 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c41fa5d42ec417aad2dde4fdad99d78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:26.402091 5893 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c41fa5d42ec417aad2dde4fdad99d78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c41fa5d42ec417aad2dde4fdad99d78, State: Initialized, Role: FOLLOWER
I20260619 16:49:26.402245 5893 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c41fa5d42ec417aad2dde4fdad99d78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c41fa5d42ec417aad2dde4fdad99d78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43367 } }
I20260619 16:49:26.402343 5893 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c41fa5d42ec417aad2dde4fdad99d78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:26.402375 5893 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c41fa5d42ec417aad2dde4fdad99d78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:26.402426 5893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c41fa5d42ec417aad2dde4fdad99d78 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:26.403095 5893 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c41fa5d42ec417aad2dde4fdad99d78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c41fa5d42ec417aad2dde4fdad99d78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43367 } }
I20260619 16:49:26.403191 5893 leader_election.cc:302] T 00000000000000000000000000000000 P 2c41fa5d42ec417aad2dde4fdad99d78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c41fa5d42ec417aad2dde4fdad99d78; no voters:
I20260619 16:49:26.403364 5893 leader_election.cc:288] T 00000000000000000000000000000000 P 2c41fa5d42ec417aad2dde4fdad99d78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:49:26.403400 5898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c41fa5d42ec417aad2dde4fdad99d78 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:26.403518 5898 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c41fa5d42ec417aad2dde4fdad99d78 [term 1 LEADER]: Becoming Leader. State: Replica: 2c41fa5d42ec417aad2dde4fdad99d78, State: Running, Role: LEADER
I20260619 16:49:26.403605 5898 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c41fa5d42ec417aad2dde4fdad99d78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c41fa5d42ec417aad2dde4fdad99d78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43367 } }
I20260619 16:49:26.403677 5893 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c41fa5d42ec417aad2dde4fdad99d78 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:26.403995 5900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c41fa5d42ec417aad2dde4fdad99d78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c41fa5d42ec417aad2dde4fdad99d78. Latest consensus state: current_term: 1 leader_uuid: "2c41fa5d42ec417aad2dde4fdad99d78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c41fa5d42ec417aad2dde4fdad99d78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43367 } } }
I20260619 16:49:26.404105 5900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c41fa5d42ec417aad2dde4fdad99d78 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:26.404982 5903 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:26.404022 5899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c41fa5d42ec417aad2dde4fdad99d78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c41fa5d42ec417aad2dde4fdad99d78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c41fa5d42ec417aad2dde4fdad99d78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43367 } } }
I20260619 16:49:26.405198 5899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c41fa5d42ec417aad2dde4fdad99d78 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:26.405570 5903 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:26.407555 5903 catalog_manager.cc:1347] Generated new cluster ID: 2cd05416ace3471d994e7fd0b1e057d0
I20260619 16:49:26.407569 5897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:26.399947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41188 (local address 127.0.98.62:43367)
0619 16:49:26.400204 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:26.400213 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:26.400242 (+ 29us) server_negotiation.cc:407] Connection header received
0619 16:49:26.401898 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:26.401909 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:26.401994 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:26.402206 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:26.402466 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:26.404049 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:26.405176 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:26.405836 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:26.405964 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:26.406767 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:26.406772 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:26.406773 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:26.407011 (+ 238us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:26.407064 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:26.407085 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:26.407155 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:26.407247 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 16:49:26.407342 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":98,"threads_started":1}
I20260619 16:49:26.407620 5903 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:49:26.408810 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:49:26.421921 5903 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:49:26.422883 5903 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:26.435913 5903 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c41fa5d42ec417aad2dde4fdad99d78: Generated new TSK 0
I20260619 16:49:26.436220 5903 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:49:26.686856 5897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:26.444771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41198 (local address 127.0.98.62:43367)
0619 16:49:26.444833 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:26.444836 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:26.444904 (+ 68us) server_negotiation.cc:407] Connection header received
0619 16:49:26.444978 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:26.444979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:26.444994 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:26.445100 (+ 106us) server_negotiation.cc:226] Negotiated authn=JWT
0619 16:49:26.445292 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:26.446081 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:26.447425 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:26.447898 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:26.447988 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:26.448213 (+ 225us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 16:49:26.686561 (+238348us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 16:49:26.686769 (+ 208us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41198: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":37}
W20260619 16:49:26.686935 5897 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41198: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 16:49:26.687017 5922 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:26.444657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43367 (local address 127.0.0.1:41198)
0619 16:49:26.444823 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:26.444835 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:26.444901 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:26.445122 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:26.445124 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:26.445126 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0619 16:49:26.445207 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:26.445210 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:26.446258 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:26.446259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:26.447257 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:26.447262 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:26.448008 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:26.448015 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:26.448069 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:26.448078 (+ 9us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 16:49:26.686743 (+238665us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 16:49:26.686904 (+ 161us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43367: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":92,"threads_started":1}
I20260619 16:49:26.687361 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 5826
2026-06-19T16:49:26Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (472 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (110 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (110 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:49:26 dist-test-slave-jsw9 krb5kdc[5934](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:49:26 dist-test-slave-jsw9 krb5kdc[5934](info): set up 2 sockets
Jun 19 16:49:26 dist-test-slave-jsw9 krb5kdc[5934](info): commencing operation
krb5kdc: starting...
W20260619 16:49:27.906677 392 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:49:27 dist-test-slave-jsw9 krb5kdc[5934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:49:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:49:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:49:28.046227 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35129
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35129
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:28.143252 5947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:28.143539 5947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:28.143567 5947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:28.145020 5947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:28.145102 5947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:28.145124 5947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:28.145143 5947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:28.145159 5947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:49:28.147225 5947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:28.147588 5947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:28.147866 5947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:28.153962 5952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:28.153913 5955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:28.154064 5953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:28.154143 5947 server_base.cc:1034] running on GCE node
I20260619 16:49:28.154640 5947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:28.155066 5947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:28.156243 5947 hybrid_clock.cc:648] HybridClock initialized: now 1781887768156222 us; error 44 us; skew 500 ppm
Jun 19 16:49:28 dist-test-slave-jsw9 krb5kdc[5934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887768, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:28.159356 5947 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:49:28.159965 5947 webserver.cc:460] Webserver started at http://127.0.98.62:45567/ using document root <none> and password file <none>
I20260619 16:49:28.160226 5947 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:28.160281 5947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:28.160427 5947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:28.162139 5947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "a97737083bff40e58355675b552414cd"
format_stamp: "Formatted at 2026-06-19 16:49:28 on dist-test-slave-jsw9"
server_key: "6760f223b03f7368b180ed22e0657d38"
server_key_iv: "656d901e6d9734def061d73bd6898cf9"
server_key_version: "encryptionkey@0"
I20260619 16:49:28.162714 5947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "a97737083bff40e58355675b552414cd"
format_stamp: "Formatted at 2026-06-19 16:49:28 on dist-test-slave-jsw9"
server_key: "6760f223b03f7368b180ed22e0657d38"
server_key_iv: "656d901e6d9734def061d73bd6898cf9"
server_key_version: "encryptionkey@0"
I20260619 16:49:28.164345 5947 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 16:49:28.165359 5962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:28.165627 5947 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:28.165747 5947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "a97737083bff40e58355675b552414cd"
format_stamp: "Formatted at 2026-06-19 16:49:28 on dist-test-slave-jsw9"
server_key: "6760f223b03f7368b180ed22e0657d38"
server_key_iv: "656d901e6d9734def061d73bd6898cf9"
server_key_version: "encryptionkey@0"
I20260619 16:49:28.165830 5947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:28.179203 5947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:28.181646 5947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:28.181859 5947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:28.187680 5947 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35129
I20260619 16:49:28.187724 6014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35129 every 8 connection(s)
I20260619 16:49:28.188326 5947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:28.189200 6015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:28.192833 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 5947
I20260619 16:49:28.192994 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:49:28.193230 392 external_mini_cluster.cc:1428] Setting key 4d4ad8099a1559429baac708ca4f5712
I20260619 16:49:28.193420 6015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a97737083bff40e58355675b552414cd: Bootstrap starting.
I20260619 16:49:28.194442 6015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a97737083bff40e58355675b552414cd: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:28.194813 6015 log.cc:824] T 00000000000000000000000000000000 P a97737083bff40e58355675b552414cd: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:28.195780 6015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a97737083bff40e58355675b552414cd: No bootstrap required, opened a new log
I20260619 16:49:28.198359 6015 raft_consensus.cc:348] T 00000000000000000000000000000000 P a97737083bff40e58355675b552414cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a97737083bff40e58355675b552414cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35129 } }
I20260619 16:49:28.198580 6015 raft_consensus.cc:374] T 00000000000000000000000000000000 P a97737083bff40e58355675b552414cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:28.198606 6015 raft_consensus.cc:729] T 00000000000000000000000000000000 P a97737083bff40e58355675b552414cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97737083bff40e58355675b552414cd, State: Initialized, Role: FOLLOWER
I20260619 16:49:28.198770 6015 consensus_queue.cc:260] T 00000000000000000000000000000000 P a97737083bff40e58355675b552414cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a97737083bff40e58355675b552414cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35129 } }
I20260619 16:49:28.198874 6015 raft_consensus.cc:388] T 00000000000000000000000000000000 P a97737083bff40e58355675b552414cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:28.198922 6015 raft_consensus.cc:482] T 00000000000000000000000000000000 P a97737083bff40e58355675b552414cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:28.198972 6015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a97737083bff40e58355675b552414cd [term 0 FOLLOWER]: Advancing to term 1
Jun 19 16:49:28 dist-test-slave-jsw9 krb5kdc[5934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:28.206569 6018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:28.193957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53402 (local address 127.0.98.62:35129)
0619 16:49:28.194157 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:28.194166 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:28.194224 (+ 58us) server_negotiation.cc:407] Connection header received
0619 16:49:28.194812 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:28.194823 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:28.194905 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:28.195058 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:28.195918 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:28.196928 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:28.198075 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:28.198700 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:28.198814 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:28.201897 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:28.201904 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:28.201905 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:28.201928 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:28.204192 (+ 2264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:28.204922 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:28.204927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:28.204927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:28.205064 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:28.205360 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:28.205362 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:28.205362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:28.205739 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:28.205883 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:28.206062 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 16:49:28.206282 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":84,"threads_started":1}
I20260619 16:49:28.217810 6015 raft_consensus.cc:504] T 00000000000000000000000000000000 P a97737083bff40e58355675b552414cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a97737083bff40e58355675b552414cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35129 } }
I20260619 16:49:28.218086 6015 leader_election.cc:302] T 00000000000000000000000000000000 P a97737083bff40e58355675b552414cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a97737083bff40e58355675b552414cd; no voters:
I20260619 16:49:28.218427 6015 leader_election.cc:288] T 00000000000000000000000000000000 P a97737083bff40e58355675b552414cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:49:28.218500 6020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a97737083bff40e58355675b552414cd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:28.218695 6020 raft_consensus.cc:686] T 00000000000000000000000000000000 P a97737083bff40e58355675b552414cd [term 1 LEADER]: Becoming Leader. State: Replica: a97737083bff40e58355675b552414cd, State: Running, Role: LEADER
I20260619 16:49:28.218811 6015 sys_catalog.cc:564] T 00000000000000000000000000000000 P a97737083bff40e58355675b552414cd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:28.218816 6020 consensus_queue.cc:237] T 00000000000000000000000000000000 P a97737083bff40e58355675b552414cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a97737083bff40e58355675b552414cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35129 } }
I20260619 16:49:28.219399 6022 sys_catalog.cc:455] T 00000000000000000000000000000000 P a97737083bff40e58355675b552414cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader a97737083bff40e58355675b552414cd. Latest consensus state: current_term: 1 leader_uuid: "a97737083bff40e58355675b552414cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a97737083bff40e58355675b552414cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35129 } } }
I20260619 16:49:28.219521 6022 sys_catalog.cc:458] T 00000000000000000000000000000000 P a97737083bff40e58355675b552414cd [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:28.219424 6021 sys_catalog.cc:455] T 00000000000000000000000000000000 P a97737083bff40e58355675b552414cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a97737083bff40e58355675b552414cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a97737083bff40e58355675b552414cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35129 } } }
I20260619 16:49:28.219817 6021 sys_catalog.cc:458] T 00000000000000000000000000000000 P a97737083bff40e58355675b552414cd [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:28.220052 6029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:28.220914 6029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:28.223290 6029 catalog_manager.cc:1347] Generated new cluster ID: f2f5e962d57a43a69a2f07aca3a1aa94
I20260619 16:49:28.223362 6029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:49:28.232336 6029 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:49:28.233071 6029 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:28.242942 6029 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a97737083bff40e58355675b552414cd: Generated new TSK 0
I20260619 16:49:28.243221 6029 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:49:28.279312 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35129
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:28.382125 6041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:28.382478 6041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:28.382520 6041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:28.384011 6041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:28.384087 6041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:28.384135 6041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:49:28.386036 6041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:28.386382 6041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:28.386646 6041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:28.392518 6047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:28.392522 6046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:28.392674 6049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:28.392802 6041 server_base.cc:1034] running on GCE node
I20260619 16:49:28.393158 6041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:28.393443 6041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:28.394665 6041 hybrid_clock.cc:648] HybridClock initialized: now 1781887768394629 us; error 66 us; skew 500 ppm
Jun 19 16:49:28 dist-test-slave-jsw9 krb5kdc[5934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887768, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:28.401715 6041 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:49:28.402410 6041 webserver.cc:460] Webserver started at http://127.0.98.1:39151/ using document root <none> and password file <none>
I20260619 16:49:28.402653 6041 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:28.402711 6041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:28.402851 6041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:28.404155 6041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-0/data/instance:
uuid: "f68d4f40d5c74225bb53f9ed73f098a3"
format_stamp: "Formatted at 2026-06-19 16:49:28 on dist-test-slave-jsw9"
server_key: "adfd210fd52860eff060a29b67d791c0"
server_key_iv: "be873914fd4aebbdaa73c2bd4e901e48"
server_key_version: "encryptionkey@0"
I20260619 16:49:28.404559 6041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f68d4f40d5c74225bb53f9ed73f098a3"
format_stamp: "Formatted at 2026-06-19 16:49:28 on dist-test-slave-jsw9"
server_key: "adfd210fd52860eff060a29b67d791c0"
server_key_iv: "be873914fd4aebbdaa73c2bd4e901e48"
server_key_version: "encryptionkey@0"
I20260619 16:49:28.406236 6041 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 16:49:28.407243 6056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:28.407509 6041 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:49:28.407610 6041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-0/wal
uuid: "f68d4f40d5c74225bb53f9ed73f098a3"
format_stamp: "Formatted at 2026-06-19 16:49:28 on dist-test-slave-jsw9"
server_key: "adfd210fd52860eff060a29b67d791c0"
server_key_iv: "be873914fd4aebbdaa73c2bd4e901e48"
server_key_version: "encryptionkey@0"
I20260619 16:49:28.407704 6041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:28.419121 6041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:28.422617 6041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:28.422814 6041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:28.423174 6041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:28.423658 6041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:28.423702 6041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:28.423738 6041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:28.423763 6041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:28.431280 6041 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38497
I20260619 16:49:28.431396 6169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38497 every 8 connection(s)
I20260619 16:49:28.431753 6041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:49:28.436841 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 6041
I20260619 16:49:28.437007 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:49:28.437206 392 external_mini_cluster.cc:1428] Setting key 87d70b25ff024ac5da4a88b14dfdbbea
Jun 19 16:49:28 dist-test-slave-jsw9 krb5kdc[5934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887768, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:28.443584 6018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:28.432823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51461 (local address 127.0.98.62:35129)
0619 16:49:28.432917 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:28.432920 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:28.433326 (+ 406us) server_negotiation.cc:407] Connection header received
0619 16:49:28.434534 (+ 1208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:28.434536 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:28.434550 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:28.434616 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:28.435487 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:28.436146 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:28.437527 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:28.437807 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:28.437910 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:28.440530 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:28.440532 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:28.440532 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:28.440540 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:28.442058 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:28.442557 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:28.442559 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:28.442559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:28.442619 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:28.442859 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:28.442860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:28.442860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:28.443001 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:28.443061 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:28.443363 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 16:49:28.443476 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 16:49:28.443785 6172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:28.432723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35129 (local address 127.0.98.1:51461)
0619 16:49:28.433010 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:28.433141 (+ 131us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:28.434395 (+ 1254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:28.434757 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:28.434762 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:28.435112 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:28.435366 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:28.435371 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:28.436262 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:28.436263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:28.437411 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:28.437416 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:28.437912 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:28.437913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:28.438092 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:28.438777 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:28.438787 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:28.440421 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:28.442151 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:28.442163 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:28.442166 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:28.442482 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:28.442675 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:28.442676 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:28.442677 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:28.442794 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:28.443056 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:28.443063 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:28.443228 (+ 165us) client_negotiation.cc:769] Sending connection context
0619 16:49:28.443319 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 16:49:28.443541 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":135,"threads_started":1}
I20260619 16:49:28.444561 6170 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35129
I20260619 16:49:28.444707 6170 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:28.444963 6170 heartbeater.cc:511] Master 127.0.98.62:35129 requested a full tablet report, sending...
I20260619 16:49:28.445614 5979 ts_manager.cc:194] Registered new tserver with Master: f68d4f40d5c74225bb53f9ed73f098a3 (127.0.98.1:38497)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 16:49:28.446379 5979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51461
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:49:28.470330 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35129
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:28.573766 6175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:28.574030 6175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:28.574067 6175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:28.575464 6175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:28.575515 6175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:28.575555 6175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:49:28.577199 6175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:28.577636 6175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:28.577951 6175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:28.584507 6183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:28.584507 6181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:28.584944 6180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:28.585067 6175 server_base.cc:1034] running on GCE node
I20260619 16:49:28.585361 6175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:28.585727 6175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:28.586898 6175 hybrid_clock.cc:648] HybridClock initialized: now 1781887768586871 us; error 44 us; skew 500 ppm
Jun 19 16:49:28 dist-test-slave-jsw9 krb5kdc[5934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887768, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:28.590587 6175 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:49:28.591144 6175 webserver.cc:460] Webserver started at http://127.0.98.2:46153/ using document root <none> and password file <none>
I20260619 16:49:28.591390 6175 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:28.591440 6175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:28.591566 6175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:28.592818 6175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-1/data/instance:
uuid: "a9d13db4f2724ea3b73fda23dcb63041"
format_stamp: "Formatted at 2026-06-19 16:49:28 on dist-test-slave-jsw9"
server_key: "5f2d26f35d79873c9eb0ada91f68842e"
server_key_iv: "bb324151e7f1efaf00ae05060145ddcd"
server_key_version: "encryptionkey@0"
I20260619 16:49:28.593197 6175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a9d13db4f2724ea3b73fda23dcb63041"
format_stamp: "Formatted at 2026-06-19 16:49:28 on dist-test-slave-jsw9"
server_key: "5f2d26f35d79873c9eb0ada91f68842e"
server_key_iv: "bb324151e7f1efaf00ae05060145ddcd"
server_key_version: "encryptionkey@0"
I20260619 16:49:28.594942 6175 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260619 16:49:28.596189 6190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:28.596441 6175 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:28.596544 6175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-1/wal
uuid: "a9d13db4f2724ea3b73fda23dcb63041"
format_stamp: "Formatted at 2026-06-19 16:49:28 on dist-test-slave-jsw9"
server_key: "5f2d26f35d79873c9eb0ada91f68842e"
server_key_iv: "bb324151e7f1efaf00ae05060145ddcd"
server_key_version: "encryptionkey@0"
I20260619 16:49:28.596628 6175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:28.607482 6175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:28.610275 6175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:28.610477 6175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:28.610756 6175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:28.611115 6175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:28.611152 6175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:28.611207 6175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:28.611230 6175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:28.617724 6175 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33543
I20260619 16:49:28.617789 6303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33543 every 8 connection(s)
I20260619 16:49:28.618332 6175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 16:49:28 dist-test-slave-jsw9 krb5kdc[5934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887768, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:28.627344 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 6175
I20260619 16:49:28.627437 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:49:28.627624 392 external_mini_cluster.cc:1428] Setting key 75070cd97753ad16b49a87833542ae04
I20260619 16:49:28.629199 6018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:28.619312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51761 (local address 127.0.98.62:35129)
0619 16:49:28.619417 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:28.619420 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:28.619651 (+ 231us) server_negotiation.cc:407] Connection header received
0619 16:49:28.620800 (+ 1149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:28.620802 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:28.620815 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:28.620879 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:28.621609 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:28.622237 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:28.623340 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:28.623554 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:28.623625 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:28.626342 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:28.626344 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:28.626344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:28.626353 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:28.627681 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:28.628149 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:28.628150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:28.628151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:28.628226 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:28.628448 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:28.628449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:28.628450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:28.628599 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:28.628659 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:28.629043 (+ 384us) server_negotiation.cc:299] Negotiation successful
0619 16:49:28.629118 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 16:49:28.629491 6306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:28.619264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35129 (local address 127.0.98.2:51761)
0619 16:49:28.619490 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:28.619539 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:28.620661 (+ 1122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:28.620948 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:28.620952 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:28.621260 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:28.621496 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:28.621501 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:28.622321 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:28.622322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:28.623231 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:28.623234 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:28.623761 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:28.623764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:28.623965 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:28.624625 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:28.624636 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:28.626225 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:28.627803 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:28.627810 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:28.627812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:28.628084 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:28.628291 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:28.628291 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:28.628292 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:28.628402 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:28.628661 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:28.628667 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:28.628904 (+ 237us) client_negotiation.cc:769] Sending connection context
0619 16:49:28.629027 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 16:49:28.629229 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":103,"threads_started":1}
I20260619 16:49:28.630127 6304 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35129
I20260619 16:49:28.630280 6304 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:28.630514 6304 heartbeater.cc:511] Master 127.0.98.62:35129 requested a full tablet report, sending...
I20260619 16:49:28.630997 5979 ts_manager.cc:194] Registered new tserver with Master: a9d13db4f2724ea3b73fda23dcb63041 (127.0.98.2:33543)
I20260619 16:49:28.631422 5979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51761
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:49:28.658039 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35129
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:28.757920 6309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:28.758260 6309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:28.758301 6309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:28.759899 6309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:28.759984 6309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:28.760020 6309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:49:28.761997 6309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:28.762343 6309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:28.762612 6309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:28.768255 6315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:28.768255 6317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:28.768369 6309 server_base.cc:1034] running on GCE node
W20260619 16:49:28.768255 6314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:28.768769 6309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:28.769063 6309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:28.770233 6309 hybrid_clock.cc:648] HybridClock initialized: now 1781887768770220 us; error 31 us; skew 500 ppm
Jun 19 16:49:28 dist-test-slave-jsw9 krb5kdc[5934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887768, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:28.773202 6309 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:49:28.773789 6309 webserver.cc:460] Webserver started at http://127.0.98.3:39555/ using document root <none> and password file <none>
I20260619 16:49:28.774045 6309 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:28.774099 6309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:28.774276 6309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:28.775520 6309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-2/data/instance:
uuid: "a5aa7cbc1bb947a6b6c4d7665d72afe6"
format_stamp: "Formatted at 2026-06-19 16:49:28 on dist-test-slave-jsw9"
server_key: "f1b0c4ff2d387a437fe3f9c34004f696"
server_key_iv: "3e71ab23b1213b7ebdb4c49ce2520360"
server_key_version: "encryptionkey@0"
I20260619 16:49:28.775894 6309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a5aa7cbc1bb947a6b6c4d7665d72afe6"
format_stamp: "Formatted at 2026-06-19 16:49:28 on dist-test-slave-jsw9"
server_key: "f1b0c4ff2d387a437fe3f9c34004f696"
server_key_iv: "3e71ab23b1213b7ebdb4c49ce2520360"
server_key_version: "encryptionkey@0"
I20260619 16:49:28.777323 6309 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 16:49:28.778152 6324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:28.778373 6309 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:28.778463 6309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-2/wal
uuid: "a5aa7cbc1bb947a6b6c4d7665d72afe6"
format_stamp: "Formatted at 2026-06-19 16:49:28 on dist-test-slave-jsw9"
server_key: "f1b0c4ff2d387a437fe3f9c34004f696"
server_key_iv: "3e71ab23b1213b7ebdb4c49ce2520360"
server_key_version: "encryptionkey@0"
I20260619 16:49:28.778532 6309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:28.790680 6309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:28.792925 6309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:28.793082 6309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:28.793352 6309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:28.793691 6309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:28.793725 6309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:28.793758 6309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:28.793777 6309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:28.800595 6309 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34795
I20260619 16:49:28.800639 6437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34795 every 8 connection(s)
I20260619 16:49:28.801010 6309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:49:28.804572 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 6309
I20260619 16:49:28.804672 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:49:28.804855 392 external_mini_cluster.cc:1428] Setting key db9aeed50712506955c9d3e96a2edcbc
Jun 19 16:49:28 dist-test-slave-jsw9 krb5kdc[5934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887768, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:28.811606 6018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:28.801849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47049 (local address 127.0.98.62:35129)
0619 16:49:28.801928 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:28.801931 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:28.802169 (+ 238us) server_negotiation.cc:407] Connection header received
0619 16:49:28.803330 (+ 1161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:28.803331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:28.803347 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:28.803407 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:28.804160 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:28.804739 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:28.805986 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:28.806220 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:28.806290 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:28.808669 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:28.808671 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:28.808671 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:28.808679 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:28.810139 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:28.810625 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:28.810626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:28.810627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:28.810700 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:28.811000 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:28.811001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:28.811001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:28.811122 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:28.811175 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:28.811452 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 16:49:28.811525 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 16:49:28.811921 6440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:28.801800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35129 (local address 127.0.98.3:47049)
0619 16:49:28.802028 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:28.802073 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:28.803201 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:28.803500 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:28.803505 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:28.803821 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:28.804067 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:28.804071 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:28.804848 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:28.804849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:28.805909 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:28.805912 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:28.806323 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:28.806325 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:28.806465 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:28.807046 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:28.807055 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:28.808574 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:28.810274 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:28.810282 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:28.810284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:28.810556 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:28.810777 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:28.810778 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:28.810778 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:28.810934 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:28.811174 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:28.811179 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:28.811329 (+ 150us) client_negotiation.cc:769] Sending connection context
0619 16:49:28.811433 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 16:49:28.811604 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":136,"threads_started":1}
I20260619 16:49:28.812434 6438 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35129
I20260619 16:49:28.812546 6438 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:28.812755 6438 heartbeater.cc:511] Master 127.0.98.62:35129 requested a full tablet report, sending...
I20260619 16:49:28.813261 5979 ts_manager.cc:194] Registered new tserver with Master: a5aa7cbc1bb947a6b6c4d7665d72afe6 (127.0.98.3:34795)
I20260619 16:49:28.813674 5979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47049
I20260619 16:49:28.817052 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/krb5kdc/test-admin.keytab.
Jun 19 16:49:28 dist-test-slave-jsw9 krb5kdc[5934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:28.835774 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 16:49:28.837658 6445 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:28.836837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50920 (local address 127.0.98.1:38497)
0619 16:49:28.837015 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:28.837021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:28.837031 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:49:28.837060 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:28.837062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:28.837077 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:28.837150 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:28.837562 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50920: BlockingRecv error: recv got EOF from 127.0.0.1:50920 (error 108)
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":93,"threads_started":1}
W20260619 16:49:28.837726 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/krb5kdc/krb5cc' of type FILE
Jun 19 16:49:28 dist-test-slave-jsw9 krb5kdc[5934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:28.839053 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 16:49:28 dist-test-slave-jsw9 krb5kdc[5934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:28.840093 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 16:49:28 dist-test-slave-jsw9 krb5kdc[5934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 16:49:28.849084 6445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:28.840802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50934 (local address 127.0.98.1:38497)
0619 16:49:28.840863 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:28.840866 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:28.840890 (+ 24us) server_negotiation.cc:407] Connection header received
0619 16:49:28.840948 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:28.840949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:28.840963 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:28.841002 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:28.841443 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:28.842347 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:28.843081 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:28.843465 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:28.843520 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:28.845933 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:28.845947 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:28.845947 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:28.845955 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:28.847835 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:28.848250 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:28.848253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:28.848254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:28.848313 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:28.848480 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:28.848481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:28.848481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:28.848644 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:28.848795 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:28.848871 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 16:49:28.848966 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260619 16:49:28.851063 6445 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:28.850514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50942 (local address 127.0.98.1:38497)
0619 16:49:28.850574 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:28.850576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:28.850634 (+ 58us) server_negotiation.cc:407] Connection header received
0619 16:49:28.850678 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:28.850678 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:28.850688 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:28.850733 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:28.851017 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50942: BlockingRecv error: recv got EOF from 127.0.0.1:50942 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260619 16:49:28.851145 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/krb5kdc/krb5cc' of type FILE
Jun 19 16:49:28 dist-test-slave-jsw9 krb5kdc[5934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:28.852290 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 16:49:28 dist-test-slave-jsw9 krb5kdc[5934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:28.853267 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 16:49:28 dist-test-slave-jsw9 krb5kdc[5934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 16:49:28.860998 6445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:28.854002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50950 (local address 127.0.98.1:38497)
0619 16:49:28.854056 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:28.854058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:28.854067 (+ 9us) server_negotiation.cc:407] Connection header received
0619 16:49:28.854116 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:28.854116 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:28.854128 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:28.854174 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:28.854708 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:28.855271 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:28.856039 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:28.856263 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:28.856317 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:28.858609 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:28.858611 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:28.858611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:28.858619 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:28.859961 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:28.860268 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:28.860269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:28.860270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:28.860317 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:28.860495 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:28.860497 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:28.860497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:28.860633 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:28.860756 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:28.860847 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 16:49:28.860920 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260619 16:49:28.862780 6445 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:28.862292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50962 (local address 127.0.98.1:38497)
0619 16:49:28.862345 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:28.862347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:28.862356 (+ 9us) server_negotiation.cc:407] Connection header received
0619 16:49:28.862378 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:28.862379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:28.862389 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:28.862451 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:28.862721 (+ 270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50962: BlockingRecv error: recv got EOF from 127.0.0.1:50962 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260619 16:49:28.863000 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781887741903642-392-0/krb5kdc/krb5cc)
Jun 19 16:49:28 dist-test-slave-jsw9 krb5kdc[5934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:28.864079 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 16:49:28 dist-test-slave-jsw9 krb5kdc[5934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:28.864975 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 16:49:28 dist-test-slave-jsw9 krb5kdc[5934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 16:49:28.872637 6445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:28.865614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50970 (local address 127.0.98.1:38497)
0619 16:49:28.865677 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:28.865679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:28.865688 (+ 9us) server_negotiation.cc:407] Connection header received
0619 16:49:28.865764 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:28.865765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:28.865776 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:28.865817 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:28.866323 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:28.866904 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:28.867640 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:28.867836 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:28.867895 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:28.870286 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:28.870288 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:28.870288 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:28.870298 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:28.871567 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:28.871911 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:28.871913 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:28.871913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:28.871972 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:28.872128 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:28.872129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:28.872130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:28.872243 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:28.872365 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:28.872436 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 16:49:28.872523 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 16:49:28.873354 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 6041
I20260619 16:49:28.878675 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 6175
I20260619 16:49:28.883704 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 6309
I20260619 16:49:28.888674 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 5947
2026-06-19T16:49:28Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1981 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:49:28 dist-test-slave-jsw9 krb5kdc[6458](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:49:28 dist-test-slave-jsw9 krb5kdc[6458](info): set up 2 sockets
Jun 19 16:49:28 dist-test-slave-jsw9 krb5kdc[6458](info): commencing operation
krb5kdc: starting...
W20260619 16:49:29.886756 392 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:49:29 dist-test-slave-jsw9 krb5kdc[6458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:49:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:49:29Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260619 16:49:30.040198 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36917
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:30.153169 6471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:30.153498 6471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:30.153541 6471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:30.155439 6471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:30.155545 6471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:30.155570 6471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:30.155588 6471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:30.155606 6471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:49:30.157819 6471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:30.158226 6471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:30.158543 6471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:30.165316 6477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:30.165400 6479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:30.165386 6476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:30.165782 6471 server_base.cc:1034] running on GCE node
I20260619 16:49:30.166064 6471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:30.166438 6471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:30.167616 6471 hybrid_clock.cc:648] HybridClock initialized: now 1781887770167590 us; error 38 us; skew 500 ppm
Jun 19 16:49:30 dist-test-slave-jsw9 krb5kdc[6458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887770, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:30.171478 6471 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260619 16:49:30.172120 6471 webserver.cc:460] Webserver started at http://127.0.98.62:40993/ using document root <none> and password file <none>
I20260619 16:49:30.172377 6471 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:30.172433 6471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:30.172573 6471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:30.173833 6471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "ae2a474c825646c2ab5e77aa28a5941a"
format_stamp: "Formatted at 2026-06-19 16:49:30 on dist-test-slave-jsw9"
server_key: "a3a736193582d575b12ccc24fb1fb07f"
server_key_iv: "53513d700e6c4e2e8b0bf31fa915c1e7"
server_key_version: "encryptionkey@0"
I20260619 16:49:30.174269 6471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "ae2a474c825646c2ab5e77aa28a5941a"
format_stamp: "Formatted at 2026-06-19 16:49:30 on dist-test-slave-jsw9"
server_key: "a3a736193582d575b12ccc24fb1fb07f"
server_key_iv: "53513d700e6c4e2e8b0bf31fa915c1e7"
server_key_version: "encryptionkey@0"
I20260619 16:49:30.175863 6471 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:30.177258 6486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:30.177785 6471 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:30.177917 6471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "ae2a474c825646c2ab5e77aa28a5941a"
format_stamp: "Formatted at 2026-06-19 16:49:30 on dist-test-slave-jsw9"
server_key: "a3a736193582d575b12ccc24fb1fb07f"
server_key_iv: "53513d700e6c4e2e8b0bf31fa915c1e7"
server_key_version: "encryptionkey@0"
I20260619 16:49:30.178015 6471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:30.216365 6471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:30.219583 6471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:30.219775 6471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:30.225967 6538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36917 every 8 connection(s)
I20260619 16:49:30.226039 6471 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36917
I20260619 16:49:30.227051 6471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:30.228350 6539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:30.229135 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 6471
I20260619 16:49:30.229275 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:49:30.229488 392 external_mini_cluster.cc:1428] Setting key 898d1c331fa8ff5f9b06e60ed1359a55
I20260619 16:49:30.232133 6539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a: Bootstrap starting.
I20260619 16:49:30.233053 6539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:30.233448 6539 log.cc:824] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:30.234472 6539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a: No bootstrap required, opened a new log
I20260619 16:49:30.237265 6539 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } }
I20260619 16:49:30.237483 6539 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:30.237520 6539 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae2a474c825646c2ab5e77aa28a5941a, State: Initialized, Role: FOLLOWER
I20260619 16:49:30.237697 6539 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } }
I20260619 16:49:30.237780 6539 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:30.237808 6539 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:30.237862 6539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [term 0 FOLLOWER]: Advancing to term 1
Jun 19 16:49:30 dist-test-slave-jsw9 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 16:49:30.238901 6539 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } }
I20260619 16:49:30.239048 6539 leader_election.cc:302] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae2a474c825646c2ab5e77aa28a5941a; no voters:
I20260619 16:49:30.239285 6539 leader_election.cc:288] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:49:30.239382 6544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:30.239609 6544 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [term 1 LEADER]: Becoming Leader. State: Replica: ae2a474c825646c2ab5e77aa28a5941a, State: Running, Role: LEADER
I20260619 16:49:30.239660 6539 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:30.239720 6544 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } }
I20260619 16:49:30.240384 6545 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae2a474c825646c2ab5e77aa28a5941a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } }
I20260619 16:49:30.240562 6545 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:30.240967 6553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:30.241211 6546 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae2a474c825646c2ab5e77aa28a5941a. Latest consensus state: current_term: 1 leader_uuid: "ae2a474c825646c2ab5e77aa28a5941a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } }
I20260619 16:49:30.241284 6546 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:30.241567 6553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:30.243625 6553 catalog_manager.cc:1347] Generated new cluster ID: d85fac49974d47e09e4548a836db6c9a
I20260619 16:49:30.243685 6553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:49:30.244083 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:30.230276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33520 (local address 127.0.98.62:36917)
0619 16:49:30.230532 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:30.230544 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:30.230584 (+ 40us) server_negotiation.cc:407] Connection header received
0619 16:49:30.231222 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:30.231234 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:30.231314 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:30.231518 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:30.232241 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:30.233744 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:30.234749 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:30.235428 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:30.235571 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:30.238714 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:30.238721 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:30.238722 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:30.238740 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:30.241437 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:30.241925 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:30.241929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:30.241930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:30.242041 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:30.242333 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:30.242334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:30.242335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:30.242704 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:30.242881 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:30.243509 (+ 628us) server_negotiation.cc:299] Negotiation successful
0619 16:49:30.243904 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":106,"threads_started":1}
I20260619 16:49:30.253027 6553 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:49:30.253732 6553 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:30.259759 6553 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a: Generated new TSK 0
I20260619 16:49:30.260062 6553 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260619 16:49:30.304191 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36917
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:30.412572 6565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:30.412878 6565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:30.412923 6565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:30.414482 6565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:30.414547 6565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:30.414592 6565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:49:30.416733 6565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:30.417122 6565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:30.417445 6565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:30.424046 6573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:30.424119 6570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:30.424180 6571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:30.424441 6565 server_base.cc:1034] running on GCE node
I20260619 16:49:30.424690 6565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:30.425210 6565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:30.426406 6565 hybrid_clock.cc:648] HybridClock initialized: now 1781887770426369 us; error 63 us; skew 500 ppm
Jun 19 16:49:30 dist-test-slave-jsw9 krb5kdc[6458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887770, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:30.430043 6565 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 16:49:30.430686 6565 webserver.cc:460] Webserver started at http://127.0.98.1:37547/ using document root <none> and password file <none>
I20260619 16:49:30.430932 6565 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:30.430989 6565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:30.431142 6565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:30.432556 6565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-0/data/instance:
uuid: "830c2952c1c0432082d86c11093a1a1d"
format_stamp: "Formatted at 2026-06-19 16:49:30 on dist-test-slave-jsw9"
server_key: "40ea35f2df39d60df2dabf381b93f6c3"
server_key_iv: "8099ccb9de34d7c1ac67a9aab0209cc9"
server_key_version: "encryptionkey@0"
I20260619 16:49:30.432984 6565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-0/wal/instance:
uuid: "830c2952c1c0432082d86c11093a1a1d"
format_stamp: "Formatted at 2026-06-19 16:49:30 on dist-test-slave-jsw9"
server_key: "40ea35f2df39d60df2dabf381b93f6c3"
server_key_iv: "8099ccb9de34d7c1ac67a9aab0209cc9"
server_key_version: "encryptionkey@0"
I20260619 16:49:30.434998 6565 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260619 16:49:30.436208 6580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:30.436542 6565 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:49:30.436661 6565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-0/wal
uuid: "830c2952c1c0432082d86c11093a1a1d"
format_stamp: "Formatted at 2026-06-19 16:49:30 on dist-test-slave-jsw9"
server_key: "40ea35f2df39d60df2dabf381b93f6c3"
server_key_iv: "8099ccb9de34d7c1ac67a9aab0209cc9"
server_key_version: "encryptionkey@0"
I20260619 16:49:30.436759 6565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:30.463344 6565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:30.466257 6565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:30.466487 6565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:30.467180 6565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:30.467600 6565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:30.467640 6565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:30.467679 6565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:30.467702 6565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:30.476480 6565 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38953
I20260619 16:49:30.476497 6693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38953 every 8 connection(s)
I20260619 16:49:30.477105 6565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:49:30.483331 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 6565
I20260619 16:49:30.483459 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:49:30.483698 392 external_mini_cluster.cc:1428] Setting key 6ac01fd8f513fc27d8f0951231b9dce9
Jun 19 16:49:30 dist-test-slave-jsw9 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887770, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 16:49:30.489611 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:30.478105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59953 (local address 127.0.98.62:36917)
0619 16:49:30.478244 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:30.478246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:30.478645 (+ 399us) server_negotiation.cc:407] Connection header received
0619 16:49:30.479477 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:30.479479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:30.479494 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:30.479585 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:30.480743 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:30.481313 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:30.482669 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:30.482944 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:30.483012 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:30.485965 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:30.485967 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:30.485967 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:30.485976 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:30.487553 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:30.488028 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:30.488029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:30.488029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:30.488098 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:30.488511 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:30.488512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:30.488513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:30.488672 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:30.488760 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:30.489378 (+ 618us) server_negotiation.cc:299] Negotiation successful
0619 16:49:30.489490 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 16:49:30.489838 6696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:30.478083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.98.1:59953)
0619 16:49:30.478430 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:30.478522 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:30.479366 (+ 844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:30.479661 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:30.479665 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:30.480345 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:30.480596 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:30.480602 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:30.481437 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:30.481439 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:30.482558 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:30.482562 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:30.483049 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:30.483050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:30.483208 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:30.483920 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:30.483931 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:30.485806 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:30.487665 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:30.487676 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:30.487680 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:30.487956 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:30.488188 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:30.488189 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:30.488189 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:30.488319 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:30.488895 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:30.488916 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:30.489204 (+ 288us) client_negotiation.cc:769] Sending connection context
0619 16:49:30.489331 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 16:49:30.489567 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":130,"threads_started":1}
I20260619 16:49:30.490499 6694 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36917
I20260619 16:49:30.490620 6694 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:30.490833 6694 heartbeater.cc:511] Master 127.0.98.62:36917 requested a full tablet report, sending...
I20260619 16:49:30.491590 6503 ts_manager.cc:194] Registered new tserver with Master: 830c2952c1c0432082d86c11093a1a1d (127.0.98.1:38953)
I20260619 16:49:30.492619 6503 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59953
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260619 16:49:30.519865 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36917
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:30.626159 6699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:30.626559 6699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:30.626609 6699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:30.628341 6699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:30.628432 6699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:30.628482 6699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:49:30.630510 6699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:30.630847 6699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:30.631153 6699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:30.637429 6705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:30.637470 6699 server_base.cc:1034] running on GCE node
W20260619 16:49:30.637410 6704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:30.637476 6707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:30.638110 6699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:30.638540 6699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:30.639789 6699 hybrid_clock.cc:648] HybridClock initialized: now 1781887770639749 us; error 61 us; skew 500 ppm
Jun 19 16:49:30 dist-test-slave-jsw9 krb5kdc[6458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887770, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:30.643364 6699 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 16:49:30.643966 6699 webserver.cc:460] Webserver started at http://127.0.98.2:39033/ using document root <none> and password file <none>
I20260619 16:49:30.644214 6699 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:30.644269 6699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:30.644426 6699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:30.645954 6699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-1/data/instance:
uuid: "6c278dee829942d6894a91b40527fba5"
format_stamp: "Formatted at 2026-06-19 16:49:30 on dist-test-slave-jsw9"
server_key: "38844e7ffc9f733b4f8eb77fea042742"
server_key_iv: "8cc5e7aea2bdf43ea322d1e513251551"
server_key_version: "encryptionkey@0"
I20260619 16:49:30.646663 6699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6c278dee829942d6894a91b40527fba5"
format_stamp: "Formatted at 2026-06-19 16:49:30 on dist-test-slave-jsw9"
server_key: "38844e7ffc9f733b4f8eb77fea042742"
server_key_iv: "8cc5e7aea2bdf43ea322d1e513251551"
server_key_version: "encryptionkey@0"
I20260619 16:49:30.648708 6699 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20260619 16:49:30.649721 6714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:30.650084 6699 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:30.650257 6699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-1/wal
uuid: "6c278dee829942d6894a91b40527fba5"
format_stamp: "Formatted at 2026-06-19 16:49:30 on dist-test-slave-jsw9"
server_key: "38844e7ffc9f733b4f8eb77fea042742"
server_key_iv: "8cc5e7aea2bdf43ea322d1e513251551"
server_key_version: "encryptionkey@0"
I20260619 16:49:30.650349 6699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:30.664444 6699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:30.667577 6699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:30.667766 6699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:30.668080 6699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:30.668479 6699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:30.668517 6699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:30.668551 6699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:30.668569 6699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:30.677611 6699 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34327
I20260619 16:49:30.677668 6827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34327 every 8 connection(s)
I20260619 16:49:30.678090 6699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 16:49:30 dist-test-slave-jsw9 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887770, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 16:49:30.687420 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 6699
I20260619 16:49:30.687752 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:49:30.688021 392 external_mini_cluster.cc:1428] Setting key 12ae6455d6b5591165a49d55c02e0d68
I20260619 16:49:30.690389 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:30.679307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44659 (local address 127.0.98.62:36917)
0619 16:49:30.679448 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:30.679450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:30.679592 (+ 142us) server_negotiation.cc:407] Connection header received
0619 16:49:30.680194 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:30.680196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:30.680215 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:30.680282 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:30.681640 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:30.682300 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:30.683510 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:30.683720 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:30.683782 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:30.686266 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:30.686268 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:30.686268 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:30.686276 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:30.688469 (+ 2193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:30.689110 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:30.689111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:30.689112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:30.689191 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:30.689535 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:30.689536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:30.689536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:30.689689 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:30.689756 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:30.690153 (+ 397us) server_negotiation.cc:299] Negotiation successful
0619 16:49:30.690306 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:49:30.690651 6830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:30.679236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.98.2:44659)
0619 16:49:30.679421 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:30.679474 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:30.680073 (+ 599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:30.680427 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:30.680432 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:30.681194 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:30.681497 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:30.681506 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:30.682393 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:30.682394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:30.683396 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:30.683399 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:30.683795 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:30.683795 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:30.683927 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:30.684534 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:30.684544 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:30.686124 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:30.688600 (+ 2476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:30.688616 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:30.688619 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:30.689011 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:30.689259 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:30.689260 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:30.689260 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:30.689429 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:30.689755 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:30.689762 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:30.689978 (+ 216us) client_negotiation.cc:769] Sending connection context
0619 16:49:30.690106 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 16:49:30.690360 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":82,"threads_started":1}
I20260619 16:49:30.691224 6828 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36917
I20260619 16:49:30.691362 6828 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:30.691615 6828 heartbeater.cc:511] Master 127.0.98.62:36917 requested a full tablet report, sending...
I20260619 16:49:30.692138 6503 ts_manager.cc:194] Registered new tserver with Master: 6c278dee829942d6894a91b40527fba5 (127.0.98.2:34327)
I20260619 16:49:30.692924 6503 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44659
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 16:49:30.721283 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36917
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:30.822145 6833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:30.822487 6833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:30.822517 6833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:30.824134 6833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:30.824218 6833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:30.824254 6833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:49:30.826241 6833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:30.826575 6833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:30.826866 6833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:30.832762 6839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:30.832796 6838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:30.832762 6841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:30.833107 6833 server_base.cc:1034] running on GCE node
I20260619 16:49:30.833324 6833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:30.833621 6833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:30.834877 6833 hybrid_clock.cc:648] HybridClock initialized: now 1781887770834853 us; error 32 us; skew 500 ppm
Jun 19 16:49:30 dist-test-slave-jsw9 krb5kdc[6458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887770, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:30.838408 6833 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 16:49:30.839035 6833 webserver.cc:460] Webserver started at http://127.0.98.3:36229/ using document root <none> and password file <none>
I20260619 16:49:30.839277 6833 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:30.839331 6833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:30.839483 6833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:30.840973 6833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-2/data/instance:
uuid: "9dcaa1e6a3aa4e6ab19c3e19e70cdcff"
format_stamp: "Formatted at 2026-06-19 16:49:30 on dist-test-slave-jsw9"
server_key: "63bdaca7a6fd83aead89253026f040f5"
server_key_iv: "f4c7a5af04eecaaf3c7110e939eefddf"
server_key_version: "encryptionkey@0"
I20260619 16:49:30.841388 6833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9dcaa1e6a3aa4e6ab19c3e19e70cdcff"
format_stamp: "Formatted at 2026-06-19 16:49:30 on dist-test-slave-jsw9"
server_key: "63bdaca7a6fd83aead89253026f040f5"
server_key_iv: "f4c7a5af04eecaaf3c7110e939eefddf"
server_key_version: "encryptionkey@0"
I20260619 16:49:30.842926 6833 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260619 16:49:30.843753 6848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:30.843923 6833 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 16:49:30.844038 6833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-2/wal
uuid: "9dcaa1e6a3aa4e6ab19c3e19e70cdcff"
format_stamp: "Formatted at 2026-06-19 16:49:30 on dist-test-slave-jsw9"
server_key: "63bdaca7a6fd83aead89253026f040f5"
server_key_iv: "f4c7a5af04eecaaf3c7110e939eefddf"
server_key_version: "encryptionkey@0"
I20260619 16:49:30.844106 6833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:30.856372 6833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:30.859404 6833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:30.859592 6833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:30.859884 6833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:30.860253 6833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:30.860289 6833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:30.860324 6833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:30.860348 6833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:30.868017 6961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38523 every 8 connection(s)
I20260619 16:49:30.867902 6833 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38523
I20260619 16:49:30.868747 6833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:49:30.878093 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 6833
I20260619 16:49:30.878255 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:49:30.878463 392 external_mini_cluster.cc:1428] Setting key 4997868d8cd7a98487a30f1a0cda6adf
Jun 19 16:49:30 dist-test-slave-jsw9 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887770, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 16:49:30.883399 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:30.870244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51253 (local address 127.0.98.62:36917)
0619 16:49:30.870331 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:30.870606 (+ 275us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:30.871005 (+ 399us) server_negotiation.cc:407] Connection header received
0619 16:49:30.872333 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:30.872335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:30.872352 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:30.872421 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:30.874317 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:30.875058 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:30.876396 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:30.876612 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:30.876684 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:30.880153 (+ 3469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:30.880154 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:30.880155 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:30.880166 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:30.881686 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:30.882279 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:30.882280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:30.882281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:30.882351 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:30.882639 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:30.882640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:30.882640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:30.882774 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:30.882836 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:30.883182 (+ 346us) server_negotiation.cc:299] Negotiation successful
0619 16:49:30.883285 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 16:49:30.883734 6964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:30.870169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.98.3:51253)
0619 16:49:30.870766 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:30.870826 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:30.871735 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:30.872674 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:30.872681 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:30.873752 (+ 1071us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:30.874125 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:30.874134 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:30.875218 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:30.875219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:30.876275 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:30.876280 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:30.876750 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:30.876752 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:30.876905 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:30.877607 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:30.877619 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:30.879933 (+ 2314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:30.881804 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:30.881819 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:30.881823 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:30.882161 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:30.882409 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:30.882411 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:30.882411 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:30.882570 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:30.882830 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:30.882837 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:30.883034 (+ 197us) client_negotiation.cc:769] Sending connection context
0619 16:49:30.883136 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 16:49:30.883358 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":421,"threads_started":1}
I20260619 16:49:30.884454 6962 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36917
I20260619 16:49:30.884589 6962 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:30.884825 6962 heartbeater.cc:511] Master 127.0.98.62:36917 requested a full tablet report, sending...
I20260619 16:49:30.885303 6503 ts_manager.cc:194] Registered new tserver with Master: 9dcaa1e6a3aa4e6ab19c3e19e70cdcff (127.0.98.3:38523)
I20260619 16:49:30.885761 6503 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51253
I20260619 16:49:30.891124 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260619 16:49:30.923447 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44205
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36917,127.0.98.61:44205
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:31.028472 6967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:31.028816 6967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:31.028858 6967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:31.030577 6967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:31.030659 6967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:31.030681 6967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:31.030699 6967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:31.030714 6967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:49:31.033666 6967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36917,127.0.98.61:44205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
W20260619 16:49:31.034237 6967 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:36917,127.0.98.61:44205'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 16:49:31.034580 6967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:31.034930 6967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:31.041553 6973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:31.041476 6975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:31.041473 6972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:31.042249 6967 server_base.cc:1034] running on GCE node
I20260619 16:49:31.042811 6967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:31.043238 6967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:31.044457 6967 hybrid_clock.cc:648] HybridClock initialized: now 1781887771044421 us; error 86 us; skew 500 ppm
Jun 19 16:49:31 dist-test-slave-jsw9 krb5kdc[6458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887771, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:31.048346 6967 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260619 16:49:31.049026 6967 webserver.cc:460] Webserver started at http://127.0.98.61:45919/ using document root <none> and password file <none>
I20260619 16:49:31.049357 6967 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:31.049450 6967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:31.049615 6967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:31.051065 6967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-1/data/instance:
uuid: "4a5d873b1d944dbf99baa42af4f7b2f4"
format_stamp: "Formatted at 2026-06-19 16:49:31 on dist-test-slave-jsw9"
server_key: "02a69be51b9d87afbcbdbc8163e9101f"
server_key_iv: "62734cc1764e9e2e0b773f9157e767b9"
server_key_version: "encryptionkey@0"
I20260619 16:49:31.051510 6967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-1/wal/instance:
uuid: "4a5d873b1d944dbf99baa42af4f7b2f4"
format_stamp: "Formatted at 2026-06-19 16:49:31 on dist-test-slave-jsw9"
server_key: "02a69be51b9d87afbcbdbc8163e9101f"
server_key_iv: "62734cc1764e9e2e0b773f9157e767b9"
server_key_version: "encryptionkey@0"
I20260619 16:49:31.054003 6967 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:49:31.055277 6982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:31.055631 6967 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:49:31.055812 6967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-1/wal
uuid: "4a5d873b1d944dbf99baa42af4f7b2f4"
format_stamp: "Formatted at 2026-06-19 16:49:31 on dist-test-slave-jsw9"
server_key: "02a69be51b9d87afbcbdbc8163e9101f"
server_key_iv: "62734cc1764e9e2e0b773f9157e767b9"
server_key_version: "encryptionkey@0"
I20260619 16:49:31.055994 6967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:31.074127 6967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:31.077919 6967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:31.078161 6967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:31.083536 6967 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:44205
I20260619 16:49:31.083626 7034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:44205 every 8 connection(s)
I20260619 16:49:31.084030 6967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-1/data/info.pb
I20260619 16:49:31.085186 7035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:31.087282 7035 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:49:31.091130 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 6967
Jun 19 16:49:31 dist-test-slave-jsw9 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887771, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 16:49:31 dist-test-slave-jsw9 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 16:49:31.100256 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.088282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54272 (local address 127.0.98.62:36917)
0619 16:49:31.088389 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:31.088391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:31.088641 (+ 250us) server_negotiation.cc:407] Connection header received
0619 16:49:31.089309 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:31.089311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:31.089327 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:31.089405 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:31.090667 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.091415 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.093534 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.093742 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.093806 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.096760 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:31.096763 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:31.096764 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:31.096772 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:31.098314 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.098840 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.098841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.098842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.098910 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.099189 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.099191 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.099191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.099470 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:31.099555 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:31.100016 (+ 461us) server_negotiation.cc:299] Negotiation successful
0619 16:49:31.100133 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 16:49:31.100598 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.088233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.0.1:54272)
0619 16:49:31.088485 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:31.088534 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:31.089181 (+ 647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:31.089488 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:31.089493 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:31.089835 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:31.090349 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.090361 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.091549 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.091551 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.093430 (+ 1879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.093434 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.093821 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.093822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.094003 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.094671 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:31.094683 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:31.096560 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:31.098407 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.098422 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.098426 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.098758 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.098968 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.098970 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.098970 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.099095 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.099568 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:31.099577 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:31.099810 (+ 233us) client_negotiation.cc:769] Sending connection context
0619 16:49:31.100068 (+ 258us) client_negotiation.cc:240] Negotiation successful
0619 16:49:31.100296 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":105,"threads_started":1}
I20260619 16:49:31.101398 7035 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:49:31.102674 7039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.092506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57822 (local address 127.0.98.61:44205)
0619 16:49:31.092753 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:31.092759 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:31.092878 (+ 119us) server_negotiation.cc:407] Connection header received
0619 16:49:31.092911 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:31.092914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:31.092941 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:31.093052 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:31.093945 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.094936 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.095764 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.096125 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.096195 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.099132 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:31.099139 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:31.099140 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:31.099148 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:31.101125 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.101535 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.101539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.101540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.101608 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.101809 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.101810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.101810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.102023 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:31.102240 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:31.102463 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 16:49:31.102559 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":143,"threads_started":1}
Jun 19 16:49:31 dist-test-slave-jsw9 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887771, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 16:49:31.111361 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.101735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44205 (local address 127.0.0.1:57834)
0619 16:49:31.101788 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:31.101801 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:31.101839 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:31.102128 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:31.102130 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:31.102328 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:31.102404 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.102406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.103342 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.103344 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.103919 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.103922 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.104385 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.104386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.104466 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.104969 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:31.104973 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:31.106502 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:31.109839 (+ 3337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.109846 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.109847 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.110161 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.110594 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.110599 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.110600 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.110681 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.111045 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:31.111046 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:31.111101 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 16:49:31.111163 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 16:49:31.111270 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 16:49:31.111497 7040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.101775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57834 (local address 127.0.98.61:44205)
0619 16:49:31.101989 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:31.101991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:31.101996 (+ 5us) server_negotiation.cc:407] Connection header received
0619 16:49:31.102013 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:31.102014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:31.102028 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:31.102094 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:31.102470 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.103235 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.104069 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.104293 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.104353 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.106883 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:31.106885 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:31.106885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:31.106893 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:31.109602 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.110391 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.110393 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.110393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.110482 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.110796 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.110797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.110797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.110947 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:31.111015 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:31.111236 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 16:49:31.111345 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":194,"thread_start_us":150,"threads_started":1}
I20260619 16:49:31.113722 7035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4: Bootstrap starting.
I20260619 16:49:31.114764 7035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:31.115077 7035 log.cc:824] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:31.116439 7035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4: No bootstrap required, opened a new log
I20260619 16:49:31.118353 7035 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } }
I20260619 16:49:31.118575 7035 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:31.118607 7035 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a5d873b1d944dbf99baa42af4f7b2f4, State: Initialized, Role: FOLLOWER
I20260619 16:49:31.118729 7035 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } }
I20260619 16:49:31.119100 7035 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:31.119036 7042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } } }
I20260619 16:49:31.119163 7042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 16:49:31.121214 7053 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:49:31.121342 7053 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:49:31.123441 6967 master_runner.cc:428] Detected that this master 4a5d873b1d944dbf99baa42af4f7b2f4 is joining an existing cluster
I20260619 16:49:31.123534 6967 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:44205
I20260619 16:49:31.133605 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.124922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54286 (local address 127.0.98.62:36917)
0619 16:49:31.124970 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:31.124972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:31.124981 (+ 9us) server_negotiation.cc:407] Connection header received
0619 16:49:31.125008 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:31.125009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:31.125021 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:31.125088 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:31.126290 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.126959 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.128406 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.128800 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.128904 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.130315 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:31.130317 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:31.130317 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:31.130326 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:31.131892 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.132791 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.132793 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.132793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.132863 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.133058 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.133059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.133059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.133192 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:31.133253 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:31.133406 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 16:49:31.133496 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 16:49:31.133641 7040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.124713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57846 (local address 127.0.98.61:44205)
0619 16:49:31.124764 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:31.124766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:31.124922 (+ 156us) server_negotiation.cc:407] Connection header received
0619 16:49:31.125003 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:31.125004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:31.125012 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:31.125032 (+ 20us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:31.125649 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.126497 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.129048 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.129394 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.129481 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.130974 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:31.130975 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:31.130976 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:31.130983 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:31.132451 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.132905 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.132906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.132906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.132956 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.133143 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.133144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.133145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.133268 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:31.133307 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:31.133467 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 16:49:31.133559 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":18}
I20260619 16:49:31.133841 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.124672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44205 (local address 127.0.0.1:57846)
0619 16:49:31.124847 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:31.124860 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:31.124941 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:31.125081 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:31.125082 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:31.125433 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:31.125556 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.125560 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.127055 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.127057 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.128253 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.128257 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.129499 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.129500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.129576 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.130343 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:31.130347 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:31.130874 (+ 527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:31.132525 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.132526 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.132527 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.132838 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.133012 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.133012 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.133013 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.133074 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.133344 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:31.133344 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:31.133378 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 16:49:31.133498 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 16:49:31.133623 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":84,"thread_start_us":70,"threads_started":1}
I20260619 16:49:31.133919 7060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.124676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.0.1:54286)
0619 16:49:31.124865 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:31.124875 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:31.124941 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:31.125094 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:31.125094 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:31.125424 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:31.125563 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.125564 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.127085 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.127086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.128254 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.128257 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.128941 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.128943 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.129014 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.129759 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:31.129764 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:31.130180 (+ 416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:31.132523 (+ 2343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.132524 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.132524 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.132724 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.132931 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.132932 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.132932 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.132981 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.133249 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:31.133250 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:31.133310 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 16:49:31.133372 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 16:49:31.133755 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":90,"thread_start_us":113,"threads_started":1}
Jun 19 16:49:31 dist-test-slave-jsw9 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887770, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 16:49:31.145642 7040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.136610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57854 (local address 127.0.98.61:44205)
0619 16:49:31.136653 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:31.136655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:31.136847 (+ 192us) server_negotiation.cc:407] Connection header received
0619 16:49:31.136965 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:31.136966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:31.136974 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:31.137020 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:31.137627 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.138121 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.139488 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.139733 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.139794 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.142632 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:31.142634 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:31.142634 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:31.142643 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:31.144014 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.144716 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.144718 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.144718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.144782 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.145013 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.145014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.145014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.145146 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:31.145197 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:31.145465 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 16:49:31.145555 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 16:49:31.145732 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.136537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44205 (local address 127.0.0.1:57854)
0619 16:49:31.136724 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:31.136775 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:31.136865 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:31.137027 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:31.137031 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:31.137442 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:31.137525 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.137528 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.138246 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.138248 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.139347 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.139352 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.139804 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.139806 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.139907 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.140567 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:31.140573 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:31.142523 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:31.144105 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.144118 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.144122 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.144464 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.144841 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.144842 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.144842 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.144932 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.145202 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:31.145212 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:31.145294 (+ 82us) client_negotiation.cc:769] Sending connection context
0619 16:49:31.145398 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 16:49:31.145593 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":121,"threads_started":1}
I20260619 16:49:31.146660 6503 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:44205: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 44205
}
attrs {
promote: true
}
}
dest_uuid: "ae2a474c825646c2ab5e77aa28a5941a"
cas_config_opid_index: -1
I20260619 16:49:31.147143 6503 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: true } }
I20260619 16:49:31.147804 7063 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 4a5d873b1d944dbf99baa42af4f7b2f4 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: true } } }
I20260619 16:49:31.148177 7009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:31.149243 7063 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:44205
I20260619 16:49:31.149488 7009 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [term 1 FOLLOWER]: Refusing update from remote peer ae2a474c825646c2ab5e77aa28a5941a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 16:49:31.149638 6967 master.cc:537] Master@127.0.98.61:44205 shutting down...
I20260619 16:49:31.149464 7062 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ae2a474c825646c2ab5e77aa28a5941a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: true } } }
I20260619 16:49:31.149682 7062 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:31.150122 7062 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 16:49:31.151388 6488 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260619 16:49:31.152127 6488 consensus_peers.cc:487] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a -> Peer 4a5d873b1d944dbf99baa42af4f7b2f4 (127.0.98.61:44205): Couldn't send request to peer 4a5d873b1d944dbf99baa42af4f7b2f4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 16:49:31.153281 6967 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:31.153394 6967 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:31.153451 6967 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4: stopping tablet replica
I20260619 16:49:31.156697 6967 master.cc:552] Master@127.0.98.61:44205 shutdown complete.
I20260619 16:49:31.156759 6967 master_runner.cc:305] Clearing existing system tablet
I20260619 16:49:31.156883 6967 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:31.156934 6967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:31.157585 6967 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.002s
I20260619 16:49:31.158066 7067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:31.158232 6967 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:31.158290 6967 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:31.158308 6967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-1/wal
uuid: "4a5d873b1d944dbf99baa42af4f7b2f4"
format_stamp: "Formatted at 2026-06-19 16:49:31 on dist-test-slave-jsw9"
server_key: "02a69be51b9d87afbcbdbc8163e9101f"
server_key_iv: "62734cc1764e9e2e0b773f9157e767b9"
server_key_version: "encryptionkey@0"
I20260619 16:49:31.159113 6967 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:31.161188 6967 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 16:49:31.161294 6967 log.cc:1196] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 16:49:31.161484 6967 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4: Deleting consensus metadata
I20260619 16:49:31.161613 6967 master_runner.cc:315] Copying system tablet from 127.0.98.62:36917
I20260619 16:49:31.162405 6967 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:36917
I20260619 16:49:31.170805 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.162861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54302 (local address 127.0.98.62:36917)
0619 16:49:31.162917 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:31.162920 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:31.163011 (+ 91us) server_negotiation.cc:407] Connection header received
0619 16:49:31.163102 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:31.163103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:31.163113 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:31.163164 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:31.163613 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.164194 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.165303 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.165540 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.165674 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.167881 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:31.167883 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:31.167884 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:31.167893 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:31.169394 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.169877 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.169878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.169879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.169946 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.170158 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.170159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.170159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.170422 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:31.170483 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:31.170647 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 16:49:31.170723 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 16:49:31.171096 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.162770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.0.1:54302)
0619 16:49:31.162911 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:31.162925 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:31.163033 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:31.163181 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:31.163181 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:31.163419 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:31.163489 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.163492 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.164284 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.164286 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.165183 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.165185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.165901 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.165902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.165991 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.167071 (+ 1080us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:31.167077 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:31.167611 (+ 534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:31.169498 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.169500 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.169500 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.169769 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.170015 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.170016 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.170016 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.170085 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.170484 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:31.170485 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:31.170548 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 16:49:31.170617 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 16:49:31.170753 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":77,"threads_started":1}
I20260619 16:49:31.172183 6523 tablet_copy_service.cc:140] P ae2a474c825646c2ab5e77aa28a5941a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4a5d873b1d944dbf99baa42af4f7b2f4 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:54302)
I20260619 16:49:31.172313 6523 tablet_copy_service.cc:161] P ae2a474c825646c2ab5e77aa28a5941a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4a5d873b1d944dbf99baa42af4f7b2f4 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:54302: session id = 4a5d873b1d944dbf99baa42af4f7b2f4-00000000000000000000000000000000
I20260619 16:49:31.173334 6523 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 16:49:31.174340 6967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 16:49:31.176178 6967 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4: tablet copy: Starting download of 0 data blocks...
I20260619 16:49:31.176379 6967 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4: tablet copy: Starting download of 1 WAL segments...
I20260619 16:49:31.177589 6967 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 16:49:31.178982 6523 tablet_copy_service.cc:342] P ae2a474c825646c2ab5e77aa28a5941a: Request end of tablet copy session 4a5d873b1d944dbf99baa42af4f7b2f4-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:54302
I20260619 16:49:31.179068 6523 tablet_copy_service.cc:433] P ae2a474c825646c2ab5e77aa28a5941a: ending tablet copy session 4a5d873b1d944dbf99baa42af4f7b2f4-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4a5d873b1d944dbf99baa42af4f7b2f4
I20260619 16:49:31.182608 6967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:31.184621 7075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:31.184701 7074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:31.184763 6967 server_base.cc:1034] running on GCE node
W20260619 16:49:31.184799 7077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:31.185520 6967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:31.185768 6967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:31.186909 6967 hybrid_clock.cc:648] HybridClock initialized: now 1781887771186888 us; error 40 us; skew 500 ppm
Jun 19 16:49:31 dist-test-slave-jsw9 krb5kdc[6458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887771, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:31.189373 6967 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260619 16:49:31.189775 6967 webserver.cc:460] Webserver started at http://127.0.98.61:45883/ using document root <none> and password file <none>
I20260619 16:49:31.189909 6967 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:31.189947 6967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:31.190871 6967 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260619 16:49:31.191787 7084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:31.192041 6967 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:31.192128 6967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-1/wal
uuid: "4a5d873b1d944dbf99baa42af4f7b2f4"
format_stamp: "Formatted at 2026-06-19 16:49:31 on dist-test-slave-jsw9"
server_key: "02a69be51b9d87afbcbdbc8163e9101f"
server_key_iv: "62734cc1764e9e2e0b773f9157e767b9"
server_key_version: "encryptionkey@0"
I20260619 16:49:31.192212 6967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:31.204306 6967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:31.207062 6967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:31.210995 6967 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:44205
I20260619 16:49:31.211367 7136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:44205 every 8 connection(s)
I20260619 16:49:31.211506 6967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-1/data/info.pb
I20260619 16:49:31.212371 7137 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:49:31.212805 7137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4: Bootstrap starting.
I20260619 16:49:31.215770 7137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:49:31.216050 7137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4: Bootstrap complete.
I20260619 16:49:31.216285 7137 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: true } }
I20260619 16:49:31.216372 7137 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4a5d873b1d944dbf99baa42af4f7b2f4, State: Initialized, Role: LEARNER
I20260619 16:49:31.216409 7137 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: true } }
I20260619 16:49:31.216662 7137 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:31.216620 7140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: true } } }
I20260619 16:49:31.216715 7140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [sys.catalog]: This master's current role is: LEARNER
I20260619 16:49:31.218478 7151 catalog_manager.cc:1259] Loaded cluster ID: d85fac49974d47e09e4548a836db6c9a
I20260619 16:49:31.218614 7151 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4: loading cluster ID for follower catalog manager: success
I20260619 16:49:31.219767 7151 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4: acquiring CA information for follower catalog manager: success
I20260619 16:49:31.220214 7151 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:49:31.265066 7152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.257198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57876 (local address 127.0.98.61:44205)
0619 16:49:31.257395 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:31.257397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:31.257408 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:49:31.257439 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:31.257440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:31.257454 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:31.257545 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:31.258384 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.259291 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.260175 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.260405 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.260470 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.261913 (+ 1443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:31.261914 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:31.261915 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:31.261923 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:31.263792 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.264199 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.264200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.264201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.264273 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.264449 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.264451 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.264451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.264586 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:31.264690 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:31.264809 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 16:49:31.264916 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":117,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260619 16:49:31.302590 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36871
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36917,127.0.98.61:44205,127.0.98.60:36871
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:31.407687 7155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:31.408129 7155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:31.408181 7155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:31.409847 7155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:31.409921 7155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:31.409937 7155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:31.409950 7155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:31.409961 7155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:49:31.412827 7155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36917,127.0.98.61:44205,127.0.98.60:36871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:31.413204 7155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:31.413463 7155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:31.420171 7161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:31.420187 7160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:31.420171 7163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:31.420218 7155 server_base.cc:1034] running on GCE node
I20260619 16:49:31.420881 7155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:31.421307 7155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:31.422462 7155 hybrid_clock.cc:648] HybridClock initialized: now 1781887771422447 us; error 36 us; skew 500 ppm
Jun 19 16:49:31 dist-test-slave-jsw9 krb5kdc[6458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887771, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:31.426733 7155 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260619 16:49:31.427474 7155 webserver.cc:460] Webserver started at http://127.0.98.60:42249/ using document root <none> and password file <none>
I20260619 16:49:31.427760 7155 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:31.427821 7155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:31.427976 7155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:31.429355 7155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-2/data/instance:
uuid: "bfeed1e6da914f00952279d4b079cd75"
format_stamp: "Formatted at 2026-06-19 16:49:31 on dist-test-slave-jsw9"
server_key: "238b60e5ebbd1336efa1f918259e78e0"
server_key_iv: "ebfb01ec2d9a4d9e9752c6c548f25bd5"
server_key_version: "encryptionkey@0"
I20260619 16:49:31.429831 7155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-2/wal/instance:
uuid: "bfeed1e6da914f00952279d4b079cd75"
format_stamp: "Formatted at 2026-06-19 16:49:31 on dist-test-slave-jsw9"
server_key: "238b60e5ebbd1336efa1f918259e78e0"
server_key_iv: "ebfb01ec2d9a4d9e9752c6c548f25bd5"
server_key_version: "encryptionkey@0"
I20260619 16:49:31.431694 7155 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 16:49:31.432704 7170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:31.432965 7155 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:31.433076 7155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-2/wal
uuid: "bfeed1e6da914f00952279d4b079cd75"
format_stamp: "Formatted at 2026-06-19 16:49:31 on dist-test-slave-jsw9"
server_key: "238b60e5ebbd1336efa1f918259e78e0"
server_key_iv: "ebfb01ec2d9a4d9e9752c6c548f25bd5"
server_key_version: "encryptionkey@0"
I20260619 16:49:31.433171 7155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:31.454634 7155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:31.458796 7155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:31.459022 7155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:31.467684 7155 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36871
I20260619 16:49:31.467722 7222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36871 every 8 connection(s)
I20260619 16:49:31.468242 7155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-2/data/info.pb
I20260619 16:49:31.469377 7223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:31.470824 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 7155
I20260619 16:49:31.472079 7223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:49:31 dist-test-slave-jsw9 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887771, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 16:49:31 dist-test-slave-jsw9 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 16:49:31.484658 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.473527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54314 (local address 127.0.98.62:36917)
0619 16:49:31.473587 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:31.473589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:31.473600 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:49:31.473629 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:31.473631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:31.473643 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:31.473690 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:31.474413 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.475141 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.476319 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.476605 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.476667 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.479550 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:31.479552 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:31.479552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:31.479562 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:31.482109 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.483176 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.483178 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.483178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.483318 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.483739 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.483740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.483740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.483934 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:31.484035 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:31.484435 (+ 400us) server_negotiation.cc:299] Negotiation successful
0619 16:49:31.484550 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 16:49:31.484862 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.471671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35654 (local address 127.0.98.60:36871)
0619 16:49:31.472026 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:31.472040 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:31.472089 (+ 49us) server_negotiation.cc:407] Connection header received
0619 16:49:31.472742 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:31.472752 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:31.472817 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:31.472988 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:31.473742 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.475026 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.475835 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.476388 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.476502 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.480045 (+ 3543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:31.480055 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:31.480055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:31.480064 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:31.483185 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.483648 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.483653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.483654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.483717 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.483996 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.483998 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.483998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.484256 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:31.484428 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:31.484592 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 16:49:31.484665 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":106,"threads_started":1}
I20260619 16:49:31.484862 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.473282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.0.1:54314)
0619 16:49:31.473468 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:31.473517 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:31.473582 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:31.473729 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:31.473733 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:31.474040 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:31.474152 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.474157 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.475255 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.475257 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.476218 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.476221 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.476680 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.476681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.476754 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.477454 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:31.477464 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:31.479428 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:31.482275 (+ 2847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.482282 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.482286 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.483034 (+ 748us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.483465 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.483466 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.483467 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.483625 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.484043 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:31.484051 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:31.484259 (+ 208us) client_negotiation.cc:769] Sending connection context
0619 16:49:31.484354 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 16:49:31.484558 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":108,"threads_started":1}
I20260619 16:49:31.485543 7223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:49:31 dist-test-slave-jsw9 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887771, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 16:49:31.493938 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.485875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44205 (local address 127.0.0.1:57888)
0619 16:49:31.485923 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:31.485935 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:31.485980 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:31.486159 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:31.486160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:31.486337 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:31.486400 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.486403 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.487192 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.487193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.487770 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.487772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.488239 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.488241 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.488312 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.488872 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:31.488878 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:31.490572 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:31.492499 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.492500 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.492501 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.492698 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.492971 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.492972 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.492973 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.493070 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.493468 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:31.493471 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:31.493555 (+ 84us) client_negotiation.cc:769] Sending connection context
0619 16:49:31.493714 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 16:49:31.493846 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 16:49:31.493949 6694 heartbeater.cc:503] Master 127.0.98.62:36917 was elected leader, sending a full tablet report...
I20260619 16:49:31.493996 7152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.485981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57888 (local address 127.0.98.61:44205)
0619 16:49:31.486036 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:31.486038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:31.486046 (+ 8us) server_negotiation.cc:407] Connection header received
0619 16:49:31.486074 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:31.486075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:31.486086 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:31.486141 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:31.486495 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.487105 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.487900 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.488140 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.488209 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.490713 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:31.490715 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:31.490716 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:31.490729 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:31.492245 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.492808 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.492809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.492809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.492874 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.493123 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.493124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.493124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.493260 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:31.493333 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:31.493779 (+ 446us) server_negotiation.cc:299] Negotiation successful
0619 16:49:31.493891 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 16:49:31.494428 7223 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36871 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:49:31 dist-test-slave-jsw9 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887771, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 16:49:31.503706 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.494769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36871 (local address 127.0.0.1:35660)
0619 16:49:31.494840 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:31.494853 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:31.494906 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:31.495045 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:31.495046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:31.495191 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:31.495242 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.495243 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.496217 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.496218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.496869 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.496875 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.497398 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.497400 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.497464 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.498013 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:31.498019 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:31.499866 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:31.501973 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.501975 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.501976 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.502339 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.502855 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.502856 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.502856 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.502933 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.503366 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:31.503368 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:31.503415 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 16:49:31.503491 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 16:49:31.503616 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 16:49:31.503777 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.494811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35660 (local address 127.0.98.60:36871)
0619 16:49:31.494854 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:31.494856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:31.494891 (+ 35us) server_negotiation.cc:407] Connection header received
0619 16:49:31.494987 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:31.494988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:31.494997 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:31.495042 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:31.495323 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.495994 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.497049 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.497271 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.497345 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.499994 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:31.499996 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:31.499996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:31.500005 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:31.501337 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.502486 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.502487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.502488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.502560 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.503112 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.503113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.503114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.503261 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:31.503333 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:31.503556 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 16:49:31.503679 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":25}
I20260619 16:49:31.506558 7223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75: Bootstrap starting.
I20260619 16:49:31.507843 7223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:31.508219 7223 log.cc:824] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:31.509178 7223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75: No bootstrap required, opened a new log
I20260619 16:49:31.510998 7223 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36871 } }
I20260619 16:49:31.511164 7223 raft_consensus.cc:374] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:31.511188 7223 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfeed1e6da914f00952279d4b079cd75, State: Initialized, Role: FOLLOWER
I20260619 16:49:31.511329 7223 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36871 } }
I20260619 16:49:31.511756 7223 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:31.511755 7228 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36871 } } }
I20260619 16:49:31.511868 7228 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 16:49:31.513717 7239 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:49:31.513792 7239 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:49:31.517592 7155 master_runner.cc:428] Detected that this master bfeed1e6da914f00952279d4b079cd75 is joining an existing cluster
I20260619 16:49:31.517721 7155 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:36871
I20260619 16:49:31.529138 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.519225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54330 (local address 127.0.98.62:36917)
0619 16:49:31.519319 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:31.519321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:31.519610 (+ 289us) server_negotiation.cc:407] Connection header received
0619 16:49:31.519692 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:31.519694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:31.519706 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:31.519758 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:31.520270 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.521059 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.522305 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.522612 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.522663 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.524303 (+ 1640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:31.524305 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:31.524305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:31.524313 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:31.527544 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.528199 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.528200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.528200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.528271 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.528499 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.528501 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.528501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.528659 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:31.528771 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:31.528953 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 16:49:31.529045 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 16:49:31.529332 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.519027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.0.1:54330)
0619 16:49:31.519544 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:31.519553 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:31.519635 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:31.519809 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:31.519810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:31.520084 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:31.520168 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.520171 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.521164 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.521165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.522139 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.522141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.522770 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.522771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.522838 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.523463 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:31.523469 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:31.524049 (+ 580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:31.527803 (+ 3754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.527806 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.527807 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.528097 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.528336 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.528336 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.528337 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.528406 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.528759 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:31.528760 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:31.528819 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 16:49:31.528892 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 16:49:31.529116 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":20,"thread_start_us":107,"threads_started":1}
I20260619 16:49:31.537096 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 6565
Jun 19 16:49:31 dist-test-slave-jsw9 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887770, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 16:49:31.543675 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.519411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36871 (local address 127.0.0.1:35662)
0619 16:49:31.519714 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:31.519723 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:31.519781 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:31.519923 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:31.519924 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:31.520116 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:31.520172 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.520174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.521050 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.521051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.522507 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.522510 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.523256 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.523257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.523332 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.538802 (+ 15470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:31.538810 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:31.539543 (+ 733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:31.542565 (+ 3022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.542566 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.542567 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.542781 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.542953 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.542953 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.542954 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.543015 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.543297 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:31.543298 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:31.543340 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 16:49:31.543397 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 16:49:31.543526 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":94,"thread_start_us":68,"threads_started":1}
I20260619 16:49:31.544783 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.519361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35662 (local address 127.0.98.60:36871)
0619 16:49:31.519411 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:31.519413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:31.519766 (+ 353us) server_negotiation.cc:407] Connection header received
0619 16:49:31.519834 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:31.519835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:31.519843 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:31.519868 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:31.520237 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.520961 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.522609 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.523124 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.523177 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.539696 (+ 16519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:31.539697 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:31.539698 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:31.539708 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:31.542459 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.542850 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.542851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.542851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.542900 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.543071 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.543072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.543073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.543221 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:31.543278 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:31.544490 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0619 16:49:31.544679 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 16:49:31.547194 7152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.519147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57892 (local address 127.0.98.61:44205)
0619 16:49:31.519279 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:31.519281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:31.519338 (+ 57us) server_negotiation.cc:407] Connection header received
0619 16:49:31.519488 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:31.519489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:31.519503 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:31.519579 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:31.520107 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.520962 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.522616 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.523130 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.523177 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.540504 (+ 17327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:31.540507 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:31.540507 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:31.540518 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:31.543598 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.546024 (+ 2426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.546027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.546027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.546139 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.546413 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.546414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.546415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.546612 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:31.546659 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:31.546884 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 16:49:31.547087 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260619 16:49:31.547606 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.519027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44205 (local address 127.0.0.1:57892)
0619 16:49:31.519265 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:31.519277 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:31.519383 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:31.519594 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:31.519595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:31.519923 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:31.520013 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.520016 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.521050 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.521051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.522508 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.522510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.523258 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.523259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.523321 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.539610 (+ 16289us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:31.539615 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:31.540144 (+ 529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:31.544306 (+ 4162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.544308 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.544309 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.544816 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.546266 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.546267 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.546268 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.546345 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.546706 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:31.546707 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:31.546778 (+ 71us) client_negotiation.cc:769] Sending connection context
0619 16:49:31.547328 (+ 550us) client_negotiation.cc:240] Negotiation successful
0619 16:49:31.547488 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":105,"thread_start_us":109,"threads_started":1}
I20260619 16:49:31.549338 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.532436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36871 (local address 127.0.0.1:35666)
0619 16:49:31.532547 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:31.532559 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:31.532602 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:31.532859 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:31.532860 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:31.533126 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:31.533189 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.533192 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.534355 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.534357 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.535001 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.535004 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.535409 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.535410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.535610 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.536320 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:31.536326 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:31.538896 (+ 2570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:31.547799 (+ 8903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.547802 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.547802 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.548110 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.548482 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.548483 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.548483 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.548577 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.548908 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:31.548910 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:31.548993 (+ 83us) client_negotiation.cc:769] Sending connection context
0619 16:49:31.549047 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 16:49:31.549221 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260619 16:49:31.549796 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.532502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35666 (local address 127.0.98.60:36871)
0619 16:49:31.532723 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:31.532725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:31.532731 (+ 6us) server_negotiation.cc:407] Connection header received
0619 16:49:31.532769 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:31.532770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:31.532780 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:31.532842 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:31.533283 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.534230 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.535096 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.535338 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.535401 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.539378 (+ 3977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:31.539380 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:31.539381 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:31.539390 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:31.545872 (+ 6482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.548207 (+ 2335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.548208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.548209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.548314 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.548657 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.548658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.548659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.548827 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:31.549517 (+ 690us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:31.549568 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 16:49:31.549688 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":135,"threads_started":1}
I20260619 16:49:31.550513 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38953
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36917
--webserver_port=37547
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:36917,127.0.98.61:44205
--tserver_master_addrs=127.0.98.62:36917,127.0.98.61:44205,127.0.98.60:36871 with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:49:31.550500 6503 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:36871: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "bfeed1e6da914f00952279d4b079cd75"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 36871
}
attrs {
promote: true
}
}
dest_uuid: "ae2a474c825646c2ab5e77aa28a5941a"
cas_config_opid_index: 5
I20260619 16:49:31.550879 6503 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: true } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: true } }
I20260619 16:49:31.551663 7063 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bfeed1e6da914f00952279d4b079cd75 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: true } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: true } } }
I20260619 16:49:31.551870 7197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:31.553450 7063 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:36871
I20260619 16:49:31.553476 7197 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [term 1 FOLLOWER]: Refusing update from remote peer ae2a474c825646c2ab5e77aa28a5941a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 16:49:31.554471 7062 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:31.555022 7155 master.cc:537] Master@127.0.98.60:36871 shutting down...
I20260619 16:49:31.556737 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.551548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44205 (local address 127.0.0.1:57894)
0619 16:49:31.551593 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:31.551607 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:31.551654 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:31.552750 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:31.552752 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:31.552755 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:31.552817 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.552820 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.553850 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.553852 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.555036 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.555040 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.556388 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.556390 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.556474 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.556513 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 16:49:31.556568 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 16:49:31.556643 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260619 16:49:31.557171 7152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.552449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57894 (local address 127.0.98.61:44205)
0619 16:49:31.552517 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:31.552519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:31.552530 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:49:31.552562 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:31.552563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:31.552595 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:31.552673 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:31.552888 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.553737 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.555160 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.556166 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.556252 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.556345 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:31.557023 (+ 678us) server_negotiation.cc:299] Negotiation successful
0619 16:49:31.557091 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 16:49:31.557571 7111 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [term 1 LEARNER]: Refusing update from remote peer ae2a474c825646c2ab5e77aa28a5941a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 16:49:31.557832 7197 raft_consensus.cc:858] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36871 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: true } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: true } }
I20260619 16:49:31.557871 7062 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260619 16:49:31.558137 7197 raft_consensus.cc:1565] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36871 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: true } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: true } }
I20260619 16:49:31.553836 7063 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ae2a474c825646c2ab5e77aa28a5941a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: true } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: true } } }
I20260619 16:49:31.558625 7063 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:31.558816 7111 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bfeed1e6da914f00952279d4b079cd75 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: true } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: true } } }
I20260619 16:49:31.559237 7197 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 4a5d873b1d944dbf99baa42af4f7b2f4 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: true } } }
I20260619 16:49:31.560712 7255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae2a474c825646c2ab5e77aa28a5941a. Latest consensus state: current_term: 1 leader_uuid: "ae2a474c825646c2ab5e77aa28a5941a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: true } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: true } } }
I20260619 16:49:31.560853 7255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [sys.catalog]: This master's current role is: LEARNER
I20260619 16:49:31.560724 7140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae2a474c825646c2ab5e77aa28a5941a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: true } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: true } } }
I20260619 16:49:31.561086 7140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [sys.catalog]: This master's current role is: LEARNER
I20260619 16:49:31.560973 7228 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae2a474c825646c2ab5e77aa28a5941a. Latest consensus state: current_term: 1 leader_uuid: "ae2a474c825646c2ab5e77aa28a5941a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: true } } }
I20260619 16:49:31.561117 7228 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 16:49:31.561111 7063 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a: attempting to promote NON_VOTER 4a5d873b1d944dbf99baa42af4f7b2f4 to VOTER
I20260619 16:49:31.561153 7228 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae2a474c825646c2ab5e77aa28a5941a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: true } } }
I20260619 16:49:31.561188 7228 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 16:49:31.561376 7063 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: false } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: true } }
I20260619 16:49:31.561746 7253 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae2a474c825646c2ab5e77aa28a5941a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: true } } }
I20260619 16:49:31.562119 7111 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [term 1 LEARNER]: Refusing update from remote peer ae2a474c825646c2ab5e77aa28a5941a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 16:49:31.562418 7063 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 16:49:31.562719 7253 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260619 16:49:31.563491 7063 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4a5d873b1d944dbf99baa42af4f7b2f4 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: false } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: true } } }
W20260619 16:49:31.563841 6488 consensus_peers.cc:487] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a -> Peer bfeed1e6da914f00952279d4b079cd75 (127.0.98.60:36871): Couldn't send request to peer bfeed1e6da914f00952279d4b079cd75. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 16:49:31.563772 7111 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4a5d873b1d944dbf99baa42af4f7b2f4 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: false } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: true } } }
I20260619 16:49:31.564700 7063 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ae2a474c825646c2ab5e77aa28a5941a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: false } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: true } } }
I20260619 16:49:31.564733 7062 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae2a474c825646c2ab5e77aa28a5941a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: false } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: true } } }
I20260619 16:49:31.564795 7063 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:31.564836 7062 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:31.565384 7140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae2a474c825646c2ab5e77aa28a5941a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: false } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: true } } }
I20260619 16:49:31.565582 7140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:49:31.568646 7155 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260619 16:49:31.568917 7155 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260619 16:49:31.569021 7155 tablet_replica.cc:330] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75: stopping tablet replica
I20260619 16:49:31.595748 7155 master.cc:552] Master@127.0.98.60:36871 shutdown complete.
I20260619 16:49:31.595844 7155 master_runner.cc:305] Clearing existing system tablet
I20260619 16:49:31.595988 7155 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:31.596047 7155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:31.598488 7155 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 16:49:31.599313 7261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:31.599668 7155 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:31.599807 7155 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:31.599840 7155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-2/wal
uuid: "bfeed1e6da914f00952279d4b079cd75"
format_stamp: "Formatted at 2026-06-19 16:49:31 on dist-test-slave-jsw9"
server_key: "238b60e5ebbd1336efa1f918259e78e0"
server_key_iv: "ebfb01ec2d9a4d9e9752c6c548f25bd5"
server_key_version: "encryptionkey@0"
I20260619 16:49:31.601737 7155 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:31.605293 7155 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 16:49:31.605413 7155 log.cc:1196] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 16:49:31.605667 7155 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75: Deleting consensus metadata
I20260619 16:49:31.606415 7155 master_runner.cc:315] Copying system tablet from 127.0.98.62:36917
I20260619 16:49:31.607214 7155 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:36917
I20260619 16:49:31.616143 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.607710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54344 (local address 127.0.98.62:36917)
0619 16:49:31.607774 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:31.607776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:31.607874 (+ 98us) server_negotiation.cc:407] Connection header received
0619 16:49:31.607952 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:31.607954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:31.607968 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:31.608037 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:31.608592 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.609418 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.610703 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.611072 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.611160 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.612575 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:31.612577 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:31.612578 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:31.612607 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:31.614614 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.615094 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.615095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.615096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.615169 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.615390 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.615392 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.615392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.615616 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:31.615674 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:31.615919 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 16:49:31.616028 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 16:49:31.616637 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.607606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.0.1:54344)
0619 16:49:31.607781 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:31.607793 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:31.607867 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:31.608053 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:31.608054 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:31.608373 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:31.608463 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.608466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.609527 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.609529 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.610595 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.610599 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.611178 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.611179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.611258 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.611937 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:31.611943 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:31.612460 (+ 517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:31.614736 (+ 2276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.614737 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.614738 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.615001 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.615233 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.615233 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.615234 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.615307 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.615724 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:31.615725 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:31.615781 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 16:49:31.615867 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 16:49:31.616011 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":94,"threads_started":1}
I20260619 16:49:31.617075 6523 tablet_copy_service.cc:140] P ae2a474c825646c2ab5e77aa28a5941a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer bfeed1e6da914f00952279d4b079cd75 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:54344)
I20260619 16:49:31.617174 6523 tablet_copy_service.cc:161] P ae2a474c825646c2ab5e77aa28a5941a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer bfeed1e6da914f00952279d4b079cd75 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:54344: session id = bfeed1e6da914f00952279d4b079cd75-00000000000000000000000000000000
I20260619 16:49:31.617954 6523 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 16:49:31.618752 7155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 16:49:31.620488 7155 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75: tablet copy: Starting download of 0 data blocks...
I20260619 16:49:31.620667 7155 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75: tablet copy: Starting download of 1 WAL segments...
I20260619 16:49:31.621708 7155 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 16:49:31.623090 6523 tablet_copy_service.cc:342] P ae2a474c825646c2ab5e77aa28a5941a: Request end of tablet copy session bfeed1e6da914f00952279d4b079cd75-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:54344
I20260619 16:49:31.623178 6523 tablet_copy_service.cc:433] P ae2a474c825646c2ab5e77aa28a5941a: ending tablet copy session bfeed1e6da914f00952279d4b079cd75-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer bfeed1e6da914f00952279d4b079cd75
I20260619 16:49:31.632504 7155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:31.634413 7268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:31.634513 7269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:31.640867 7155 server_base.cc:1034] running on GCE node
W20260619 16:49:31.640995 7271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:31.641395 7155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:31.641613 7155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:31.642735 7155 hybrid_clock.cc:648] HybridClock initialized: now 1781887771642712 us; error 50 us; skew 500 ppm
Jun 19 16:49:31 dist-test-slave-jsw9 krb5kdc[6458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887771, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:31.644733 7155 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260619 16:49:31.645108 7155 webserver.cc:460] Webserver started at http://127.0.98.60:37717/ using document root <none> and password file <none>
I20260619 16:49:31.645239 7155 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:31.645287 7155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:31.646304 7155 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:31.646977 7278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:31.647130 7155 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:31.647212 7155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-2/wal
uuid: "bfeed1e6da914f00952279d4b079cd75"
format_stamp: "Formatted at 2026-06-19 16:49:31 on dist-test-slave-jsw9"
server_key: "238b60e5ebbd1336efa1f918259e78e0"
server_key_iv: "ebfb01ec2d9a4d9e9752c6c548f25bd5"
server_key_version: "encryptionkey@0"
I20260619 16:49:31.647323 7155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:31.653936 7155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:31.656596 7155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:31.660610 7155 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36871
I20260619 16:49:31.661023 7330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36871 every 8 connection(s)
I20260619 16:49:31.661257 7155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/master-2/data/info.pb
I20260619 16:49:31.662544 7331 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:49:31.663301 7331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75: Bootstrap starting.
I20260619 16:49:31.666430 7331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:49:31.666687 7331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75: Bootstrap complete.
I20260619 16:49:31.666991 7331 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: false } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: true } }
I20260619 16:49:31.667107 7331 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bfeed1e6da914f00952279d4b079cd75, State: Initialized, Role: LEARNER
I20260619 16:49:31.667147 7331 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: false } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: true } }
I20260619 16:49:31.667348 7331 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:31.668820 7334 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: false } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: true } } }
I20260619 16:49:31.668963 7334 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [sys.catalog]: This master's current role is: LEARNER
I20260619 16:49:31.670059 7345 catalog_manager.cc:1259] Loaded cluster ID: d85fac49974d47e09e4548a836db6c9a
I20260619 16:49:31.670126 7345 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75: loading cluster ID for follower catalog manager: success
I20260619 16:49:31.671355 7345 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75: acquiring CA information for follower catalog manager: success
I20260619 16:49:31.671754 7345 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:49:31.694366 6828 heartbeater.cc:503] Master 127.0.98.62:36917 was elected leader, sending a full tablet report...
W20260619 16:49:31.703866 7249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:31.704202 7249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:31.704247 7249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:31.707799 7249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:31.707929 7249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:31.707990 7249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:49:31.710580 7249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38953
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37547
--tserver_master_addrs=127.0.98.62:36917,127.0.98.61:44205,127.0.98.60:36871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:31.710976 7249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:31.711265 7249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:31.719254 7350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:31.722267 7249 server_base.cc:1034] running on GCE node
W20260619 16:49:31.722875 7353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:31.724506 7351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:31.725262 7249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:31.725807 7249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:31.727022 7249 hybrid_clock.cc:648] HybridClock initialized: now 1781887771727000 us; error 43 us; skew 500 ppm
Jun 19 16:49:31 dist-test-slave-jsw9 krb5kdc[6458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887771, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:31.730300 7249 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 16:49:31.730888 7249 webserver.cc:460] Webserver started at http://127.0.98.1:37547/ using document root <none> and password file <none>
I20260619 16:49:31.731150 7249 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:31.731220 7249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:31.732931 7249 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 16:49:31.733786 7360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:31.734001 7249 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:31.734125 7249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-0/wal
uuid: "830c2952c1c0432082d86c11093a1a1d"
format_stamp: "Formatted at 2026-06-19 16:49:30 on dist-test-slave-jsw9"
server_key: "40ea35f2df39d60df2dabf381b93f6c3"
server_key_iv: "8099ccb9de34d7c1ac67a9aab0209cc9"
server_key_version: "encryptionkey@0"
I20260619 16:49:31.734617 7249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:31.751670 7249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:31.758520 7249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:31.760547 7249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:31.761374 7249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:31.762012 7249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:31.762068 7249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:31.762094 7249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:31.762106 7249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:31.772531 7249 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38953
I20260619 16:49:31.772605 7473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38953 every 8 connection(s)
I20260619 16:49:31.773343 7249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:49:31.773638 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 7249
I20260619 16:49:31.773924 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 6699
Jun 19 16:49:31 dist-test-slave-jsw9 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887771, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 19 16:49:31 dist-test-slave-jsw9 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887771, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 16:49:31.789392 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34327
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36917
--webserver_port=39033
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:36917,127.0.98.61:44205
--tserver_master_addrs=127.0.98.62:36917,127.0.98.61:44205,127.0.98.60:36871 with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:49:31 dist-test-slave-jsw9 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887771, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 16:49:31.794298 7480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.775328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51693 (local address 127.0.98.60:36871)
0619 16:49:31.775503 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:31.775505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:31.777377 (+ 1872us) server_negotiation.cc:407] Connection header received
0619 16:49:31.777484 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:31.777486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:31.777505 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:31.777693 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:31.778152 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.779361 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.781120 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.781486 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.782129 (+ 643us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.790310 (+ 8181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:31.790313 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:31.790313 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:31.790324 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:31.792537 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.792933 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.792934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.792934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.793003 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.793215 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.793217 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.793217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.793398 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:31.793448 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:31.793958 (+ 510us) server_negotiation.cc:299] Negotiation successful
0619 16:49:31.794138 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":88,"threads_started":1}
I20260619 16:49:31.794605 7478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.776789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44205 (local address 127.0.98.1:52619)
0619 16:49:31.779294 (+ 2505us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:31.779307 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:31.779415 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:31.779621 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:31.779622 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:31.779852 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:31.779916 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.779919 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.780914 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.780915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.781708 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.781710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.782278 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.782279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.782352 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.786163 (+ 3811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:31.786167 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:31.789545 (+ 3378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:31.792001 (+ 2456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.792019 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.792024 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.792714 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.792966 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.792967 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.792967 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.793146 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.793493 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:31.793501 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:31.793730 (+ 229us) client_negotiation.cc:769] Sending connection context
0619 16:49:31.793869 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 16:49:31.794300 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2482,"mutex_wait_us":6,"thread_start_us":91,"threads_started":1}
I20260619 16:49:31.794729 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.774954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36871 (local address 127.0.98.1:51693)
0619 16:49:31.777285 (+ 2331us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:31.777295 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:31.777364 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:31.777704 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:31.777705 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:31.777901 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:31.778056 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.778062 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.779541 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.779542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.781019 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.781021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.781589 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.781591 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.781771 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.786135 (+ 4364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:31.786155 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:31.790173 (+ 4018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:31.792622 (+ 2449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.792623 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.792624 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.792849 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.793073 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.793074 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.793074 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.793145 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.793503 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:31.793503 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:31.793730 (+ 227us) client_negotiation.cc:769] Sending connection context
0619 16:49:31.794471 (+ 741us) client_negotiation.cc:240] Negotiation successful
0619 16:49:31.794598 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2305,"mutex_wait_us":11,"spinlock_wait_cycles":896,"thread_start_us":434,"threads_started":1}
I20260619 16:49:31.794983 7152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.776875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52619 (local address 127.0.98.61:44205)
0619 16:49:31.776931 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:31.776934 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:31.779381 (+ 2447us) server_negotiation.cc:407] Connection header received
0619 16:49:31.779498 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:31.779500 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:31.779516 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:31.779552 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:31.780024 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.780795 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.781796 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.782100 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.782145 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.789711 (+ 7566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:31.789714 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:31.789714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:31.789725 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:31.791868 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.792824 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.792825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.792826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.792892 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.793215 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.793217 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.793217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.793404 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:31.793463 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:31.794787 (+ 1324us) server_negotiation.cc:299] Negotiation successful
0619 16:49:31.794894 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 16:49:31.795399 7476 heartbeater.cc:347] Connected to a master server at 127.0.98.61:44205
I20260619 16:49:31.795413 7474 heartbeater.cc:347] Connected to a master server at 127.0.98.60:36871
I20260619 16:49:31.795564 7474 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:31.795573 7476 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:31.795941 7476 heartbeater.cc:511] Master 127.0.98.61:44205 requested a full tablet report, sending...
I20260619 16:49:31.796078 7474 heartbeater.cc:511] Master 127.0.98.60:36871 requested a full tablet report, sending...
I20260619 16:49:31.796341 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.774762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34401 (local address 127.0.98.62:36917)
0619 16:49:31.774821 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:31.774823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:31.775845 (+ 1022us) server_negotiation.cc:407] Connection header received
0619 16:49:31.776515 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:31.776517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:31.776533 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:31.776603 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:31.778876 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.780058 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.785877 (+ 5819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.786300 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.786387 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.790679 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:31.790681 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:31.790682 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:31.790693 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:31.793601 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.794219 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.794221 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.794221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.794297 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.795292 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.795294 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.795294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.795495 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:31.795584 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:31.796153 (+ 569us) server_negotiation.cc:299] Negotiation successful
0619 16:49:31.796259 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 16:49:31.796737 7479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.774954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.98.1:34401)
0619 16:49:31.775636 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:31.775705 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:31.776392 (+ 687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:31.777227 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:31.777233 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:31.777691 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:31.778725 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.778729 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.780179 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.780180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.783583 (+ 3403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.783588 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.786476 (+ 2888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.786478 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.786570 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.787117 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:31.787123 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:31.790568 (+ 3445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:31.793714 (+ 3146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.793715 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.793716 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.793979 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.794979 (+ 1000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.794980 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.794981 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.795219 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.795613 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:31.795614 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:31.795682 (+ 68us) client_negotiation.cc:769] Sending connection context
0619 16:49:31.796443 (+ 761us) client_negotiation.cc:240] Negotiation successful
0619 16:49:31.796616 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":1,"thread_start_us":404,"threads_started":1}
I20260619 16:49:31.796718 7293 ts_manager.cc:194] Registered new tserver with Master: 830c2952c1c0432082d86c11093a1a1d (127.0.98.1:38953)
I20260619 16:49:31.796718 7101 ts_manager.cc:194] Registered new tserver with Master: 830c2952c1c0432082d86c11093a1a1d (127.0.98.1:38953)
I20260619 16:49:31.797366 7475 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36917
I20260619 16:49:31.797433 7475 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:31.797530 7475 heartbeater.cc:511] Master 127.0.98.62:36917 requested a full tablet report, sending...
I20260619 16:49:31.798122 6503 ts_manager.cc:194] Re-registered known tserver with Master: 830c2952c1c0432082d86c11093a1a1d (127.0.98.1:38953)
I20260619 16:49:31.800395 6503 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34401
I20260619 16:49:31.886929 6962 heartbeater.cc:503] Master 127.0.98.62:36917 was elected leader, sending a full tablet report...
W20260619 16:49:31.895991 7482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:31.896301 7482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:31.896343 7482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:31.897874 7482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:31.897943 7482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:31.897989 7482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:49:31.900034 7482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34327
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39033
--tserver_master_addrs=127.0.98.62:36917,127.0.98.61:44205,127.0.98.60:36871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:31.900396 7482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:31.900688 7482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:31.906931 7488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:31.906921 7487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:31.907145 7490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:31.908252 7482 server_base.cc:1034] running on GCE node
I20260619 16:49:31.908594 7482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:31.908936 7482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:31.910116 7482 hybrid_clock.cc:648] HybridClock initialized: now 1781887771910070 us; error 64 us; skew 500 ppm
Jun 19 16:49:31 dist-test-slave-jsw9 krb5kdc[6458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887771, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:31.913641 7482 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 16:49:31.914394 7482 webserver.cc:460] Webserver started at http://127.0.98.2:39033/ using document root <none> and password file <none>
I20260619 16:49:31.914678 7482 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:31.914855 7482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:31.917037 7482 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 16:49:31.917894 7497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:31.918143 7482 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:31.918313 7482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-1/wal
uuid: "6c278dee829942d6894a91b40527fba5"
format_stamp: "Formatted at 2026-06-19 16:49:30 on dist-test-slave-jsw9"
server_key: "38844e7ffc9f733b4f8eb77fea042742"
server_key_iv: "8cc5e7aea2bdf43ea322d1e513251551"
server_key_version: "encryptionkey@0"
I20260619 16:49:31.918834 7482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:31.940892 7482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:31.948442 7482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:31.948616 7482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:31.948939 7482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:31.949587 7482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:31.949654 7482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:31.949683 7482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:31.949694 7482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:31.961359 7482 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34327
I20260619 16:49:31.961431 7610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34327 every 8 connection(s)
I20260619 16:49:31.961944 7482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:49:31.969792 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 7482
I20260619 16:49:31.970002 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 6833
Jun 19 16:49:31 dist-test-slave-jsw9 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887771, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 19 16:49:31 dist-test-slave-jsw9 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887771, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 19 16:49:31 dist-test-slave-jsw9 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887771, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 16:49:31.977484 7152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.963541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54935 (local address 127.0.98.61:44205)
0619 16:49:31.963595 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:31.963597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:31.963862 (+ 265us) server_negotiation.cc:407] Connection header received
0619 16:49:31.964957 (+ 1095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:31.964959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:31.964973 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:31.965004 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:31.965911 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.966758 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.968366 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.968672 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.968744 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.973250 (+ 4506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:31.973252 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:31.973253 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:31.973263 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:31.975568 (+ 2305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.976147 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.976149 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.976149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.976220 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.976564 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.976565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.976566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.976744 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:31.976790 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:31.977252 (+ 462us) server_negotiation.cc:299] Negotiation successful
0619 16:49:31.977360 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 16:49:31.977492 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.963320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34197 (local address 127.0.98.62:36917)
0619 16:49:31.963538 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:31.963540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:31.963559 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:49:31.964777 (+ 1218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:31.964779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:31.964794 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:31.964842 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:31.965929 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.966737 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.968388 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.968621 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.968662 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.973935 (+ 5273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:31.973938 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:31.973938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:31.973948 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:31.976110 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.976467 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.976468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.976468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.976549 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.976732 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.976733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.976734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.976868 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:31.976908 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:31.977254 (+ 346us) server_negotiation.cc:299] Negotiation successful
0619 16:49:31.977360 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173}
I20260619 16:49:31.978282 7617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.963466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44205 (local address 127.0.98.2:54935)
0619 16:49:31.963686 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:31.963696 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:31.964817 (+ 1121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:31.965074 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:31.965075 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:31.965521 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:31.965804 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.965812 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.966898 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.966899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.968262 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.968265 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.968770 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.968771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.968956 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.969834 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:31.969847 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:31.973107 (+ 3260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:31.975798 (+ 2691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.975799 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.975800 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.976040 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.976293 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.976294 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.976294 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.976488 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.976836 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:31.976845 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:31.977077 (+ 232us) client_negotiation.cc:769] Sending connection context
0619 16:49:31.977664 (+ 587us) client_negotiation.cc:240] Negotiation successful
0619 16:49:31.977922 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":156,"thread_start_us":111,"threads_started":1}
I20260619 16:49:31.978282 7615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.962994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.98.2:34197)
0619 16:49:31.963250 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:31.963304 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:31.964546 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:31.964966 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:31.964977 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:31.965463 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:31.965805 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.965812 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.966868 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.966870 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.968272 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.968274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.968769 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.968770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.968956 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.969834 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:31.969847 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:31.973752 (+ 3905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:31.976198 (+ 2446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.976199 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.976199 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.976386 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.976621 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.976621 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.976622 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.976673 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.976948 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:31.976948 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:31.977077 (+ 129us) client_negotiation.cc:769] Sending connection context
0619 16:49:31.978020 (+ 943us) client_negotiation.cc:240] Negotiation successful
0619 16:49:31.978144 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":57,"thread_start_us":115,"threads_started":1}
I20260619 16:49:31.978914 7613 heartbeater.cc:347] Connected to a master server at 127.0.98.61:44205
I20260619 16:49:31.979069 7613 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:31.979290 7480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.963388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34397 (local address 127.0.98.60:36871)
0619 16:49:31.963604 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:31.963606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:31.963615 (+ 9us) server_negotiation.cc:407] Connection header received
0619 16:49:31.964860 (+ 1245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:31.964862 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:31.964883 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:31.964947 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:31.966281 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.967102 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.968426 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.968769 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.968827 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.972582 (+ 3755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:31.972586 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:31.972586 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:31.972597 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:31.974871 (+ 2274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.975644 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.975939 (+ 295us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.975940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.976024 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:31.978532 (+ 2508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:31.978534 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:31.978534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:31.978752 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:31.978806 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:31.979087 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 16:49:31.979215 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189}
I20260619 16:49:31.979339 7613 heartbeater.cc:511] Master 127.0.98.61:44205 requested a full tablet report, sending...
I20260619 16:49:31.979462 7616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:31.962994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36871 (local address 127.0.98.2:34397)
0619 16:49:31.963372 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:31.963381 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:31.964553 (+ 1172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:31.964967 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:31.964972 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:31.965480 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:31.965805 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.965812 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.967279 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.967280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.968313 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:31.968314 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:31.968841 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:31.968841 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:31.968955 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:31.970285 (+ 1330us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:31.970289 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:31.972405 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:31.974974 (+ 2569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.975002 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.975006 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.975403 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.976327 (+ 924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:31.976329 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:31.976330 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:31.977347 (+ 1017us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:31.978859 (+ 1512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:31.978860 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:31.978917 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 16:49:31.978991 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 16:49:31.979130 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":167,"thread_start_us":119,"threads_started":1}
I20260619 16:49:31.980262 7612 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36917
I20260619 16:49:31.980329 7612 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:31.980404 7612 heartbeater.cc:511] Master 127.0.98.62:36917 requested a full tablet report, sending...
I20260619 16:49:31.980489 7101 ts_manager.cc:194] Registered new tserver with Master: 6c278dee829942d6894a91b40527fba5 (127.0.98.2:34327)
I20260619 16:49:31.980813 7611 heartbeater.cc:347] Connected to a master server at 127.0.98.60:36871
I20260619 16:49:31.980878 7611 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:31.980949 7611 heartbeater.cc:511] Master 127.0.98.60:36871 requested a full tablet report, sending...
I20260619 16:49:31.980967 6503 ts_manager.cc:194] Re-registered known tserver with Master: 6c278dee829942d6894a91b40527fba5 (127.0.98.2:34327)
I20260619 16:49:31.981382 7293 ts_manager.cc:194] Registered new tserver with Master: 6c278dee829942d6894a91b40527fba5 (127.0.98.2:34327)
I20260619 16:49:31.981583 6503 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34197
I20260619 16:49:31.983613 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38523
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36917
--webserver_port=36229
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:36917,127.0.98.61:44205
--tserver_master_addrs=127.0.98.62:36917,127.0.98.61:44205,127.0.98.60:36871 with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:32.096982 7618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:32.097270 7618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:32.097308 7618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:32.098758 7618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:32.098816 7618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:32.098860 7618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:49:32.100534 7618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38523
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36229
--tserver_master_addrs=127.0.98.62:36917,127.0.98.61:44205,127.0.98.60:36871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:32.100829 7618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:32.101101 7618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:32.109912 7627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:32.109912 7625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:32.111779 7624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:32.112016 7618 server_base.cc:1034] running on GCE node
I20260619 16:49:32.112430 7618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:32.112793 7618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:32.113981 7618 hybrid_clock.cc:648] HybridClock initialized: now 1781887772113961 us; error 46 us; skew 500 ppm
Jun 19 16:49:32 dist-test-slave-jsw9 krb5kdc[6458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887772, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:32.117296 7618 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 16:49:32.117919 7618 webserver.cc:460] Webserver started at http://127.0.98.3:36229/ using document root <none> and password file <none>
I20260619 16:49:32.118173 7618 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:32.118287 7618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:32.119992 7618 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:49:32.120702 7634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:32.120868 7618 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:49:32.120954 7618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-2/wal
uuid: "9dcaa1e6a3aa4e6ab19c3e19e70cdcff"
format_stamp: "Formatted at 2026-06-19 16:49:30 on dist-test-slave-jsw9"
server_key: "63bdaca7a6fd83aead89253026f040f5"
server_key_iv: "f4c7a5af04eecaaf3c7110e939eefddf"
server_key_version: "encryptionkey@0"
I20260619 16:49:32.121392 7618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:32.132264 7618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:32.134867 7618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:32.135025 7618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:32.135311 7618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:32.135671 7618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:32.135707 7618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:32.135732 7618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:32.135754 7618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:32.145162 7480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:32.140600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35672 (local address 127.0.98.60:36871)
0619 16:49:32.140670 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:32.140672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:32.140684 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:49:32.140724 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:32.140725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:32.140758 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:32.140843 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:32.141023 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.142001 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.143824 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.144570 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.144674 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:32.144781 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:32.144980 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 16:49:32.145048 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 16:49:32.145215 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:32.140363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36871 (local address 127.0.0.1:35672)
0619 16:49:32.140507 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:32.140525 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:32.140623 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:32.140855 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:32.140856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:32.140861 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:32.140951 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:32.140954 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.142132 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.142134 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:32.143670 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:32.143677 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.144731 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.144734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:32.144842 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:32.144852 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 16:49:32.144989 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 16:49:32.145087 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260619 16:49:32.145758 7305 raft_consensus.cc:1203] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 16:49:32.145989 7334 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae2a474c825646c2ab5e77aa28a5941a. Latest consensus state: current_term: 1 leader_uuid: "ae2a474c825646c2ab5e77aa28a5941a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: false } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: true } } }
I20260619 16:49:32.146124 7334 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [sys.catalog]: This master's current role is: LEARNER
I20260619 16:49:32.147048 7618 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38523
I20260619 16:49:32.147589 7618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:49:32.148583 7748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38523 every 8 connection(s)
I20260619 16:49:32.151328 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 7618
Jun 19 16:49:32 dist-test-slave-jsw9 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887772, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 19 16:49:32 dist-test-slave-jsw9 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887772, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 16:49:32 dist-test-slave-jsw9 krb5kdc[6458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887772, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 16:49:32.167932 7152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:32.149326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38681 (local address 127.0.98.61:44205)
0619 16:49:32.149392 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:32.149394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:32.149510 (+ 116us) server_negotiation.cc:407] Connection header received
0619 16:49:32.150776 (+ 1266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:32.150778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:32.150795 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:32.150831 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:32.152798 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.153645 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.156490 (+ 2845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.156778 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.156826 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:32.159965 (+ 3139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:32.159968 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:32.159969 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:32.159990 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:32.164867 (+ 4877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:32.165785 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:32.165787 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:32.165787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:32.165886 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:32.166701 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:32.166703 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:32.166704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:32.167061 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:32.167143 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:32.167498 (+ 355us) server_negotiation.cc:299] Negotiation successful
0619 16:49:32.167611 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 16:49:32.168684 7480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:32.149165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41931 (local address 127.0.98.60:36871)
0619 16:49:32.149214 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:32.149215 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:32.149452 (+ 237us) server_negotiation.cc:407] Connection header received
0619 16:49:32.150775 (+ 1323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:32.150777 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:32.150801 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:32.150831 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:32.152763 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.153645 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.155634 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.155981 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.156054 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:32.161123 (+ 5069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:32.161125 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:32.161126 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:32.161137 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:32.164657 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:32.166292 (+ 1635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:32.166295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:32.166295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:32.166373 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:32.166706 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:32.166707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:32.166707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:32.166879 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:32.166941 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:32.168441 (+ 1500us) server_negotiation.cc:299] Negotiation successful
0619 16:49:32.168578 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 16:49:32.169430 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:32.149069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36871 (local address 127.0.98.3:41931)
0619 16:49:32.149382 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:32.149392 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:32.150612 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:32.151108 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:32.151114 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:32.151852 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:32.152579 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:32.152591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.153832 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.153833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:32.155548 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:32.155553 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.156145 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.156146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:32.156338 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:32.157521 (+ 1183us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:32.157525 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:32.160933 (+ 3408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:32.165876 (+ 4943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:32.165878 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:32.165878 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:32.166136 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:32.166442 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:32.166443 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:32.166444 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:32.166600 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:32.167026 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:32.167037 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:32.167311 (+ 274us) client_negotiation.cc:769] Sending connection context
0619 16:49:32.167446 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 16:49:32.167738 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":82,"thread_start_us":73,"threads_started":1}
I20260619 16:49:32.170171 7749 heartbeater.cc:347] Connected to a master server at 127.0.98.60:36871
I20260619 16:49:32.170367 7749 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:32.170646 7749 heartbeater.cc:511] Master 127.0.98.60:36871 requested a full tablet report, sending...
I20260619 16:49:32.169421 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:32.149239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44205 (local address 127.0.98.3:38681)
0619 16:49:32.149447 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:32.149456 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:32.150613 (+ 1157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:32.151108 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:32.151114 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:32.151852 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:32.152579 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:32.152591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.153832 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.153833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:32.156345 (+ 2512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:32.156347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.156929 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.156930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:32.157141 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:32.157512 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:32.157517 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:32.159801 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:32.165015 (+ 5214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:32.165035 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:32.165040 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:32.165666 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:32.165956 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:32.165957 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:32.165958 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:32.166596 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:32.167203 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:32.167204 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:32.167360 (+ 156us) client_negotiation.cc:769] Sending connection context
0619 16:49:32.168123 (+ 763us) client_negotiation.cc:240] Negotiation successful
0619 16:49:32.169159 (+ 1036us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":35,"thread_start_us":62,"threads_started":1}
I20260619 16:49:32.171241 7751 heartbeater.cc:347] Connected to a master server at 127.0.98.61:44205
I20260619 16:49:32.171295 7751 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:32.171324 7293 ts_manager.cc:194] Registered new tserver with Master: 9dcaa1e6a3aa4e6ab19c3e19e70cdcff (127.0.98.3:38523)
I20260619 16:49:32.171383 7751 heartbeater.cc:511] Master 127.0.98.61:44205 requested a full tablet report, sending...
I20260619 16:49:32.172214 7101 ts_manager.cc:194] Registered new tserver with Master: 9dcaa1e6a3aa4e6ab19c3e19e70cdcff (127.0.98.3:38523)
I20260619 16:49:32.173570 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:32.148697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32813 (local address 127.0.98.62:36917)
0619 16:49:32.148839 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:32.148842 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:32.149108 (+ 266us) server_negotiation.cc:407] Connection header received
0619 16:49:32.150778 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:32.150780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:32.150797 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:32.150874 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:32.153160 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.153989 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.155675 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.156009 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.156054 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:32.160559 (+ 4505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:32.160563 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:32.160563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:32.160574 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:32.163973 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:32.170484 (+ 6511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:32.170486 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:32.170486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:32.170573 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:32.172486 (+ 1913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:32.172487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:32.172488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:32.172717 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:32.172791 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:32.173253 (+ 462us) server_negotiation.cc:299] Negotiation successful
0619 16:49:32.173445 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260619 16:49:32.174073 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:32.154992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54358 (local address 127.0.98.62:36917)
0619 16:49:32.161451 (+ 6459us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:32.161454 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:32.162123 (+ 669us) server_negotiation.cc:407] Connection header received
0619 16:49:32.162152 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:32.162154 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:32.162168 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:32.162238 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:32.163446 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.164325 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.165390 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.165711 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.165776 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:32.168089 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:32.168091 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:32.168092 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:32.168103 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:32.171854 (+ 3751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:32.172449 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:32.172450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:32.172451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:32.172558 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:32.172766 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:32.172767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:32.172768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:32.172902 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:32.173001 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:32.173568 (+ 567us) server_negotiation.cc:299] Negotiation successful
0619 16:49:32.173956 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6432,"thread_start_us":75,"threads_started":1}
I20260619 16:49:32.175217 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:32.148721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.98.3:32813)
0619 16:49:32.148933 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:32.148992 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:32.150601 (+ 1609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:32.152704 (+ 2103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:32.152705 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:32.152922 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:32.153014 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:32.153017 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.154102 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.154103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:32.155548 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:32.155554 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.156157 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.156158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:32.156338 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:32.157295 (+ 957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:32.157311 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:32.160394 (+ 3083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:32.168460 (+ 8066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:32.168461 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:32.168461 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:32.169041 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:32.170652 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:32.170653 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:32.170653 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:32.170729 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:32.172896 (+ 2167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:32.172897 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:32.172963 (+ 66us) client_negotiation.cc:769] Sending connection context
0619 16:49:32.174368 (+ 1405us) client_negotiation.cc:240] Negotiation successful
0619 16:49:32.174763 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":62,"thread_start_us":99,"threads_started":1}
I20260619 16:49:32.176254 7750 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36917
I20260619 16:49:32.176319 7750 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:32.176403 7750 heartbeater.cc:511] Master 127.0.98.62:36917 requested a full tablet report, sending...
I20260619 16:49:32.176709 6502 ts_manager.cc:194] Re-registered known tserver with Master: 9dcaa1e6a3aa4e6ab19c3e19e70cdcff (127.0.98.3:38523)
I20260619 16:49:32.177269 6502 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:32813
I20260619 16:49:32.176751 6503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:49:32.180217 6503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:49:32.181205 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:32.154647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57904 (local address 127.0.98.61:44205)
0619 16:49:32.161715 (+ 7068us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:32.161718 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:32.161728 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:49:32.161758 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:32.161759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:32.161776 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:32.161857 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:32.172599 (+ 10742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.173620 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.175296 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.175596 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.175645 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:32.177201 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:32.177202 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:32.177203 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:32.177212 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:32.179619 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:32.180138 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:32.180140 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:32.180141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:32.180228 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:32.180442 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:32.180443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:32.180443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:32.180600 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:32.180697 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:32.180917 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 16:49:32.181097 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7037,"thread_start_us":103,"threads_started":1}
I20260619 16:49:32.181347 7619 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae2a474c825646c2ab5e77aa28a5941a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: false } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: true } } }
I20260619 16:49:32.181481 7619 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:32.183174 7619 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a: attempting to promote NON_VOTER bfeed1e6da914f00952279d4b079cd75 to VOTER
I20260619 16:49:32.183379 7619 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: false } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: false } }
I20260619 16:49:32.184028 7111 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [term 1 FOLLOWER]: Refusing update from remote peer ae2a474c825646c2ab5e77aa28a5941a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 16:49:32.184068 7305 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [term 1 LEARNER]: Refusing update from remote peer ae2a474c825646c2ab5e77aa28a5941a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 16:49:32.184286 7767 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 16:49:32.184402 7619 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 16:49:32.185214 7619 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, bfeed1e6da914f00952279d4b079cd75 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: false } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: false } } }
I20260619 16:49:32.185932 7111 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, bfeed1e6da914f00952279d4b079cd75 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: false } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: false } } }
I20260619 16:49:32.186554 7619 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ae2a474c825646c2ab5e77aa28a5941a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: false } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: false } } }
I20260619 16:49:32.186632 7619 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:32.186776 7619 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae2a474c825646c2ab5e77aa28a5941a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: false } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: false } } }
I20260619 16:49:32.186836 7619 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:32.186954 7767 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae2a474c825646c2ab5e77aa28a5941a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: false } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: false } } }
I20260619 16:49:32.187012 7767 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:32.187153 7305 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, bfeed1e6da914f00952279d4b079cd75 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: false } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: false } } }
I20260619 16:49:32.188498 7334 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae2a474c825646c2ab5e77aa28a5941a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: false } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: false } } }
I20260619 16:49:32.188616 7334 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:49:32.191536 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:32.155489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35682 (local address 127.0.98.60:36871)
0619 16:49:32.161379 (+ 5890us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:32.161382 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:32.174042 (+ 12660us) server_negotiation.cc:407] Connection header received
0619 16:49:32.174092 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:32.174093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:32.174156 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:32.174310 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:32.180922 (+ 6612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.182645 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.183972 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.184318 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.184365 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:32.186406 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:32.186407 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:32.186408 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:32.186418 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:32.189411 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:32.190475 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:32.190478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:32.190478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:32.190577 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:32.190784 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:32.190785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:32.190786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:32.190955 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:32.191056 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:32.191270 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 16:49:32.191394 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5860,"thread_start_us":70,"threads_started":1}
I20260619 16:49:32.194806 7777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae2a474c825646c2ab5e77aa28a5941a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: false } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: false } } }
I20260619 16:49:32.195003 7777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:49:32.199970 7781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:32.194722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38953 (local address 127.0.0.1:47988)
0619 16:49:32.195340 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:32.195350 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:32.195417 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:32.195638 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:32.195639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:32.195641 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:32.195707 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:32.195710 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.197103 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.197104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:32.198384 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:32.198388 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.199648 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.199649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:32.199723 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:32.199731 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 16:49:32.199784 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 16:49:32.199847 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":86,"threads_started":1}
I20260619 16:49:32.200150 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:32.193948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34327 (local address 127.0.0.1:49776)
0619 16:49:32.194159 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:32.194168 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:32.194241 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:32.195964 (+ 1723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:32.195965 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:32.195967 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:32.196043 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:32.196045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.197823 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.197824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:32.198934 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:32.198937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.199954 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.199955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:32.200012 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:32.200017 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:49:32.200045 (+ 28us) client_negotiation.cc:240] Negotiation successful
0619 16:49:32.200091 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":149,"threads_started":1}
I20260619 16:49:32.200366 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:32.195004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47988 (local address 127.0.98.1:38953)
0619 16:49:32.195181 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:32.195187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:32.195484 (+ 297us) server_negotiation.cc:407] Connection header received
0619 16:49:32.195534 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:32.195536 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:32.195550 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:32.195621 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:32.195792 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.196958 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.198494 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.199426 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.199503 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:32.199607 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:32.200208 (+ 601us) server_negotiation.cc:299] Negotiation successful
0619 16:49:32.200271 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":101,"threads_started":1}
I20260619 16:49:32.200759 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:32.194394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49776 (local address 127.0.98.2:34327)
0619 16:49:32.195774 (+ 1380us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:32.195779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:32.195790 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:49:32.195830 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:32.195834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:32.195856 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:32.196122 (+ 266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:32.196195 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.197350 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.199043 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.199854 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.200486 (+ 632us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:32.200593 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:32.200622 (+ 29us) server_negotiation.cc:299] Negotiation successful
0619 16:49:32.200670 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1327,"thread_start_us":137,"threads_started":1}
I20260619 16:49:32.200953 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:32.193677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38523 (local address 127.0.0.1:56532)
0619 16:49:32.193758 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:32.193774 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:32.193838 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:32.194729 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:32.194730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:32.194732 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:32.194811 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:32.194814 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.196596 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.196597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:32.198728 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:32.198731 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.200682 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.200683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:32.200770 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:32.200775 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:49:32.200809 (+ 34us) client_negotiation.cc:240] Negotiation successful
0619 16:49:32.200871 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 16:49:32.201764 7779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:32.194284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56532 (local address 127.0.98.3:38523)
0619 16:49:32.194510 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:32.194518 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:32.194531 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:49:32.194586 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:32.194589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:32.194611 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:32.194885 (+ 274us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:32.194962 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.196411 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.198885 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.199980 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.201381 (+ 1401us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:32.201512 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:32.201594 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 16:49:32.201661 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":120,"threads_started":1}
I20260619 16:49:32.201689 7408 tablet_service.cc:1467] Processing CreateTablet for tablet 086cb14238e245deb35d6390701ea2a8 (DEFAULT_TABLE table=test-table [id=b1b29323686146e2aa3ee86d76460949]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:32.202070 7408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086cb14238e245deb35d6390701ea2a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:32.202826 7682 tablet_service.cc:1467] Processing CreateTablet for tablet 086cb14238e245deb35d6390701ea2a8 (DEFAULT_TABLE table=test-table [id=b1b29323686146e2aa3ee86d76460949]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:32.203264 7682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086cb14238e245deb35d6390701ea2a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:32.203349 7543 tablet_service.cc:1467] Processing CreateTablet for tablet 086cb14238e245deb35d6390701ea2a8 (DEFAULT_TABLE table=test-table [id=b1b29323686146e2aa3ee86d76460949]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:32.203704 7543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086cb14238e245deb35d6390701ea2a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:32.207000 7783 tablet_bootstrap.cc:492] T 086cb14238e245deb35d6390701ea2a8 P 6c278dee829942d6894a91b40527fba5: Bootstrap starting.
I20260619 16:49:32.207643 7785 tablet_bootstrap.cc:492] T 086cb14238e245deb35d6390701ea2a8 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff: Bootstrap starting.
I20260619 16:49:32.207870 7783 tablet_bootstrap.cc:654] T 086cb14238e245deb35d6390701ea2a8 P 6c278dee829942d6894a91b40527fba5: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:32.208381 7783 log.cc:824] T 086cb14238e245deb35d6390701ea2a8 P 6c278dee829942d6894a91b40527fba5: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:32.208473 7785 tablet_bootstrap.cc:654] T 086cb14238e245deb35d6390701ea2a8 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:32.208811 7785 log.cc:824] T 086cb14238e245deb35d6390701ea2a8 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:32.209194 7784 tablet_bootstrap.cc:492] T 086cb14238e245deb35d6390701ea2a8 P 830c2952c1c0432082d86c11093a1a1d: Bootstrap starting.
I20260619 16:49:32.209314 7783 tablet_bootstrap.cc:492] T 086cb14238e245deb35d6390701ea2a8 P 6c278dee829942d6894a91b40527fba5: No bootstrap required, opened a new log
I20260619 16:49:32.209396 7783 ts_tablet_manager.cc:1397] T 086cb14238e245deb35d6390701ea2a8 P 6c278dee829942d6894a91b40527fba5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:49:32.209645 7785 tablet_bootstrap.cc:492] T 086cb14238e245deb35d6390701ea2a8 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff: No bootstrap required, opened a new log
I20260619 16:49:32.209714 7785 ts_tablet_manager.cc:1397] T 086cb14238e245deb35d6390701ea2a8 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:49:32.209867 7784 tablet_bootstrap.cc:654] T 086cb14238e245deb35d6390701ea2a8 P 830c2952c1c0432082d86c11093a1a1d: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:32.210167 7784 log.cc:824] T 086cb14238e245deb35d6390701ea2a8 P 830c2952c1c0432082d86c11093a1a1d: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:32.210963 7784 tablet_bootstrap.cc:492] T 086cb14238e245deb35d6390701ea2a8 P 830c2952c1c0432082d86c11093a1a1d: No bootstrap required, opened a new log
I20260619 16:49:32.211033 7784 ts_tablet_manager.cc:1397] T 086cb14238e245deb35d6390701ea2a8 P 830c2952c1c0432082d86c11093a1a1d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:49:32.212491 7783 raft_consensus.cc:348] T 086cb14238e245deb35d6390701ea2a8 P 6c278dee829942d6894a91b40527fba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcaa1e6a3aa4e6ab19c3e19e70cdcff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38523 } } peers { permanent_uuid: "6c278dee829942d6894a91b40527fba5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34327 } } peers { permanent_uuid: "830c2952c1c0432082d86c11093a1a1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38953 } }
I20260619 16:49:32.212707 7783 raft_consensus.cc:374] T 086cb14238e245deb35d6390701ea2a8 P 6c278dee829942d6894a91b40527fba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:32.212742 7783 raft_consensus.cc:729] T 086cb14238e245deb35d6390701ea2a8 P 6c278dee829942d6894a91b40527fba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c278dee829942d6894a91b40527fba5, State: Initialized, Role: FOLLOWER
I20260619 16:49:32.213066 7783 consensus_queue.cc:260] T 086cb14238e245deb35d6390701ea2a8 P 6c278dee829942d6894a91b40527fba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcaa1e6a3aa4e6ab19c3e19e70cdcff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38523 } } peers { permanent_uuid: "6c278dee829942d6894a91b40527fba5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34327 } } peers { permanent_uuid: "830c2952c1c0432082d86c11093a1a1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38953 } }
I20260619 16:49:32.213191 7784 raft_consensus.cc:348] T 086cb14238e245deb35d6390701ea2a8 P 830c2952c1c0432082d86c11093a1a1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcaa1e6a3aa4e6ab19c3e19e70cdcff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38523 } } peers { permanent_uuid: "6c278dee829942d6894a91b40527fba5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34327 } } peers { permanent_uuid: "830c2952c1c0432082d86c11093a1a1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38953 } }
I20260619 16:49:32.213352 7784 raft_consensus.cc:374] T 086cb14238e245deb35d6390701ea2a8 P 830c2952c1c0432082d86c11093a1a1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:32.213382 7784 raft_consensus.cc:729] T 086cb14238e245deb35d6390701ea2a8 P 830c2952c1c0432082d86c11093a1a1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830c2952c1c0432082d86c11093a1a1d, State: Initialized, Role: FOLLOWER
I20260619 16:49:32.213413 7783 ts_tablet_manager.cc:1428] T 086cb14238e245deb35d6390701ea2a8 P 6c278dee829942d6894a91b40527fba5: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 16:49:32.213536 7784 consensus_queue.cc:260] T 086cb14238e245deb35d6390701ea2a8 P 830c2952c1c0432082d86c11093a1a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcaa1e6a3aa4e6ab19c3e19e70cdcff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38523 } } peers { permanent_uuid: "6c278dee829942d6894a91b40527fba5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34327 } } peers { permanent_uuid: "830c2952c1c0432082d86c11093a1a1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38953 } }
I20260619 16:49:32.213605 7785 raft_consensus.cc:348] T 086cb14238e245deb35d6390701ea2a8 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcaa1e6a3aa4e6ab19c3e19e70cdcff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38523 } } peers { permanent_uuid: "6c278dee829942d6894a91b40527fba5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34327 } } peers { permanent_uuid: "830c2952c1c0432082d86c11093a1a1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38953 } }
I20260619 16:49:32.213765 7785 raft_consensus.cc:374] T 086cb14238e245deb35d6390701ea2a8 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:32.213793 7785 raft_consensus.cc:729] T 086cb14238e245deb35d6390701ea2a8 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dcaa1e6a3aa4e6ab19c3e19e70cdcff, State: Initialized, Role: FOLLOWER
I20260619 16:49:32.213800 7784 ts_tablet_manager.cc:1428] T 086cb14238e245deb35d6390701ea2a8 P 830c2952c1c0432082d86c11093a1a1d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 16:49:32.213819 7612 heartbeater.cc:503] Master 127.0.98.62:36917 was elected leader, sending a full tablet report...
I20260619 16:49:32.213897 7785 consensus_queue.cc:260] T 086cb14238e245deb35d6390701ea2a8 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcaa1e6a3aa4e6ab19c3e19e70cdcff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38523 } } peers { permanent_uuid: "6c278dee829942d6894a91b40527fba5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34327 } } peers { permanent_uuid: "830c2952c1c0432082d86c11093a1a1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38953 } }
I20260619 16:49:32.214151 7785 ts_tablet_manager.cc:1428] T 086cb14238e245deb35d6390701ea2a8 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260619 16:49:32.214752 7750 heartbeater.cc:503] Master 127.0.98.62:36917 was elected leader, sending a full tablet report...
I20260619 16:49:32.215029 7475 heartbeater.cc:503] Master 127.0.98.62:36917 was elected leader, sending a full tablet report...
W20260619 16:49:32.277563 7477 tablet.cc:2369] T 086cb14238e245deb35d6390701ea2a8 P 830c2952c1c0432082d86c11093a1a1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:49:32.398775 7752 tablet.cc:2369] T 086cb14238e245deb35d6390701ea2a8 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:49:32.450764 7790 raft_consensus.cc:482] T 086cb14238e245deb35d6390701ea2a8 P 830c2952c1c0432082d86c11093a1a1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:32.450937 7790 raft_consensus.cc:504] T 086cb14238e245deb35d6390701ea2a8 P 830c2952c1c0432082d86c11093a1a1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcaa1e6a3aa4e6ab19c3e19e70cdcff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38523 } } peers { permanent_uuid: "6c278dee829942d6894a91b40527fba5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34327 } } peers { permanent_uuid: "830c2952c1c0432082d86c11093a1a1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38953 } }
I20260619 16:49:32.451400 7790 leader_election.cc:288] T 086cb14238e245deb35d6390701ea2a8 P 830c2952c1c0432082d86c11093a1a1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dcaa1e6a3aa4e6ab19c3e19e70cdcff (127.0.98.3:38523), 6c278dee829942d6894a91b40527fba5 (127.0.98.2:34327)
I20260619 16:49:32.454918 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:32.451849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34327 (local address 127.0.98.1:51649)
0619 16:49:32.452023 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:32.452032 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:32.452085 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:32.452275 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:32.452276 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:32.452279 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:32.452373 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:32.452375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.453039 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.453041 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:32.453972 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:32.454002 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.454633 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.454634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:32.454696 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:32.454701 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:49:32.454752 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 16:49:32.454802 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":78,"thread_start_us":102,"threads_started":1}
I20260619 16:49:32.454958 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:32.452081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51649 (local address 127.0.98.2:34327)
0619 16:49:32.452132 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:32.452134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:32.452142 (+ 8us) server_negotiation.cc:407] Connection header received
0619 16:49:32.452174 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:32.452175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:32.452206 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:32.452261 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:32.452456 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.452948 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.454109 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.454547 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.454614 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:32.454629 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:32.454811 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 16:49:32.454879 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 16:49:32.455003 7779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:32.451869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59217 (local address 127.0.98.3:38523)
0619 16:49:32.451938 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:32.451941 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:32.451956 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:49:32.451996 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:32.451997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:32.452010 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:32.452176 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:32.452352 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.452936 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.454076 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.454547 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.454614 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:32.454629 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:32.454853 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 16:49:32.454910 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 16:49:32.455168 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:32.451641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38523 (local address 127.0.98.1:59217)
0619 16:49:32.451794 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:32.451807 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:32.451880 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:32.452182 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:32.452183 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:32.452185 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:32.452279 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:32.452282 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.453042 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.453043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:32.453973 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:32.453976 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.454630 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.454631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:32.454700 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:32.454703 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:49:32.454845 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 16:49:32.454894 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":42,"thread_start_us":77,"threads_started":1}
I20260619 16:49:32.455555 7703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086cb14238e245deb35d6390701ea2a8" candidate_uuid: "830c2952c1c0432082d86c11093a1a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dcaa1e6a3aa4e6ab19c3e19e70cdcff" is_pre_election: true
I20260619 16:49:32.455559 7565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086cb14238e245deb35d6390701ea2a8" candidate_uuid: "830c2952c1c0432082d86c11093a1a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c278dee829942d6894a91b40527fba5" is_pre_election: true
I20260619 16:49:32.455752 7565 raft_consensus.cc:2454] T 086cb14238e245deb35d6390701ea2a8 P 6c278dee829942d6894a91b40527fba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 830c2952c1c0432082d86c11093a1a1d in term 0.
I20260619 16:49:32.455752 7703 raft_consensus.cc:2454] T 086cb14238e245deb35d6390701ea2a8 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 830c2952c1c0432082d86c11093a1a1d in term 0.
I20260619 16:49:32.456102 7362 leader_election.cc:302] T 086cb14238e245deb35d6390701ea2a8 P 830c2952c1c0432082d86c11093a1a1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 830c2952c1c0432082d86c11093a1a1d, 9dcaa1e6a3aa4e6ab19c3e19e70cdcff; no voters:
I20260619 16:49:32.456284 7790 raft_consensus.cc:2781] T 086cb14238e245deb35d6390701ea2a8 P 830c2952c1c0432082d86c11093a1a1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:49:32.456357 7790 raft_consensus.cc:482] T 086cb14238e245deb35d6390701ea2a8 P 830c2952c1c0432082d86c11093a1a1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:49:32.456373 7790 raft_consensus.cc:3037] T 086cb14238e245deb35d6390701ea2a8 P 830c2952c1c0432082d86c11093a1a1d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:32.457283 7790 raft_consensus.cc:504] T 086cb14238e245deb35d6390701ea2a8 P 830c2952c1c0432082d86c11093a1a1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcaa1e6a3aa4e6ab19c3e19e70cdcff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38523 } } peers { permanent_uuid: "6c278dee829942d6894a91b40527fba5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34327 } } peers { permanent_uuid: "830c2952c1c0432082d86c11093a1a1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38953 } }
I20260619 16:49:32.457432 7790 leader_election.cc:288] T 086cb14238e245deb35d6390701ea2a8 P 830c2952c1c0432082d86c11093a1a1d [CANDIDATE]: Term 1 election: Requested vote from peers 9dcaa1e6a3aa4e6ab19c3e19e70cdcff (127.0.98.3:38523), 6c278dee829942d6894a91b40527fba5 (127.0.98.2:34327)
I20260619 16:49:32.457687 7703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086cb14238e245deb35d6390701ea2a8" candidate_uuid: "830c2952c1c0432082d86c11093a1a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dcaa1e6a3aa4e6ab19c3e19e70cdcff"
I20260619 16:49:32.457728 7565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086cb14238e245deb35d6390701ea2a8" candidate_uuid: "830c2952c1c0432082d86c11093a1a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c278dee829942d6894a91b40527fba5"
I20260619 16:49:32.457787 7703 raft_consensus.cc:3037] T 086cb14238e245deb35d6390701ea2a8 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:32.457854 7565 raft_consensus.cc:3037] T 086cb14238e245deb35d6390701ea2a8 P 6c278dee829942d6894a91b40527fba5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:32.458761 7703 raft_consensus.cc:2454] T 086cb14238e245deb35d6390701ea2a8 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 830c2952c1c0432082d86c11093a1a1d in term 1.
I20260619 16:49:32.458767 7565 raft_consensus.cc:2454] T 086cb14238e245deb35d6390701ea2a8 P 6c278dee829942d6894a91b40527fba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 830c2952c1c0432082d86c11093a1a1d in term 1.
I20260619 16:49:32.458947 7362 leader_election.cc:302] T 086cb14238e245deb35d6390701ea2a8 P 830c2952c1c0432082d86c11093a1a1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c278dee829942d6894a91b40527fba5, 830c2952c1c0432082d86c11093a1a1d; no voters:
I20260619 16:49:32.459059 7790 raft_consensus.cc:2781] T 086cb14238e245deb35d6390701ea2a8 P 830c2952c1c0432082d86c11093a1a1d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:32.459230 7790 raft_consensus.cc:686] T 086cb14238e245deb35d6390701ea2a8 P 830c2952c1c0432082d86c11093a1a1d [term 1 LEADER]: Becoming Leader. State: Replica: 830c2952c1c0432082d86c11093a1a1d, State: Running, Role: LEADER
I20260619 16:49:32.459340 7790 consensus_queue.cc:237] T 086cb14238e245deb35d6390701ea2a8 P 830c2952c1c0432082d86c11093a1a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcaa1e6a3aa4e6ab19c3e19e70cdcff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38523 } } peers { permanent_uuid: "6c278dee829942d6894a91b40527fba5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34327 } } peers { permanent_uuid: "830c2952c1c0432082d86c11093a1a1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38953 } }
I20260619 16:49:32.460680 6502 catalog_manager.cc:5495] T 086cb14238e245deb35d6390701ea2a8 P 830c2952c1c0432082d86c11093a1a1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 830c2952c1c0432082d86c11093a1a1d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "830c2952c1c0432082d86c11093a1a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcaa1e6a3aa4e6ab19c3e19e70cdcff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c278dee829942d6894a91b40527fba5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "830c2952c1c0432082d86c11093a1a1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38953 } health_report { overall_health: HEALTHY } } }
W20260619 16:49:32.463239 7614 tablet.cc:2369] T 086cb14238e245deb35d6390701ea2a8 P 6c278dee829942d6894a91b40527fba5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:49:32.489109 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:32.486670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47998 (local address 127.0.98.1:38953)
0619 16:49:32.486739 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:32.486741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:32.486751 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:49:32.486801 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:32.486802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:32.486828 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:32.486894 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:32.487046 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.487553 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.488290 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.488494 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.488557 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:32.488708 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:32.488845 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:32.488931 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:32.488970 (+ 39us) server_negotiation.cc:299] Negotiation successful
0619 16:49:32.489004 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 16:49:32.491647 7703 raft_consensus.cc:1261] T 086cb14238e245deb35d6390701ea2a8 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff [term 1 FOLLOWER]: Refusing update from remote peer 830c2952c1c0432082d86c11093a1a1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:32.491652 7565 raft_consensus.cc:1261] T 086cb14238e245deb35d6390701ea2a8 P 6c278dee829942d6894a91b40527fba5 [term 1 FOLLOWER]: Refusing update from remote peer 830c2952c1c0432082d86c11093a1a1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:32.492187 7790 consensus_queue.cc:1035] T 086cb14238e245deb35d6390701ea2a8 P 830c2952c1c0432082d86c11093a1a1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c278dee829942d6894a91b40527fba5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:32.492345 7794 consensus_queue.cc:1035] T 086cb14238e245deb35d6390701ea2a8 P 830c2952c1c0432082d86c11093a1a1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dcaa1e6a3aa4e6ab19c3e19e70cdcff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:32.496517 7799 mvcc.cc:205] Tried to move back new op lower bound from 7298612316121817088 to 7298612315994316800. Current Snapshot: MvccSnapshot[applied={T|T < 7298612316121817088}]
I20260619 16:49:32.497083 7797 mvcc.cc:205] Tried to move back new op lower bound from 7298612316121817088 to 7298612315994316800. Current Snapshot: MvccSnapshot[applied={T|T < 7298612316121817088 or (T in {7298612316121817088})}]
I20260619 16:49:32.497967 7798 mvcc.cc:205] Tried to move back new op lower bound from 7298612316121817088 to 7298612315994316800. Current Snapshot: MvccSnapshot[applied={T|T < 7298612316121817088 or (T in {7298612316121817088})}]
I20260619 16:49:32.499601 6502 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:32.499725 6502 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:32.501358 6502 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae2a474c825646c2ab5e77aa28a5941a: Sending DeleteTablet for 3 replicas of tablet 086cb14238e245deb35d6390701ea2a8
I20260619 16:49:32.501968 7408 tablet_service.cc:1514] Processing DeleteTablet for tablet 086cb14238e245deb35d6390701ea2a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:32 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47988
I20260619 16:49:32.503321 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 6471
I20260619 16:49:32.503391 7682 tablet_service.cc:1514] Processing DeleteTablet for tablet 086cb14238e245deb35d6390701ea2a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:32 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56532
I20260619 16:49:32.504798 7543 tablet_service.cc:1514] Processing DeleteTablet for tablet 086cb14238e245deb35d6390701ea2a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:32 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49776
I20260619 16:49:32.509457 7809 tablet_replica.cc:330] T 086cb14238e245deb35d6390701ea2a8 P 6c278dee829942d6894a91b40527fba5: stopping tablet replica
I20260619 16:49:32.509652 7809 raft_consensus.cc:2229] T 086cb14238e245deb35d6390701ea2a8 P 6c278dee829942d6894a91b40527fba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:32.509799 7809 raft_consensus.cc:2258] T 086cb14238e245deb35d6390701ea2a8 P 6c278dee829942d6894a91b40527fba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:32.510478 7809 ts_tablet_manager.cc:1905] T 086cb14238e245deb35d6390701ea2a8 P 6c278dee829942d6894a91b40527fba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:32.513182 7809 ts_tablet_manager.cc:1918] T 086cb14238e245deb35d6390701ea2a8 P 6c278dee829942d6894a91b40527fba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:32.513293 7809 log.cc:1196] T 086cb14238e245deb35d6390701ea2a8 P 6c278dee829942d6894a91b40527fba5: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-1/wal/wals/086cb14238e245deb35d6390701ea2a8
I20260619 16:49:32.513579 7809 ts_tablet_manager.cc:1939] T 086cb14238e245deb35d6390701ea2a8 P 6c278dee829942d6894a91b40527fba5: Deleting consensus metadata
I20260619 16:49:32.513609 7808 tablet_replica.cc:330] T 086cb14238e245deb35d6390701ea2a8 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff: stopping tablet replica
I20260619 16:49:32.513761 7808 raft_consensus.cc:2229] T 086cb14238e245deb35d6390701ea2a8 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:32.513919 7808 raft_consensus.cc:2258] T 086cb14238e245deb35d6390701ea2a8 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 16:49:32.514036 7500 connection.cc:441] server connection from 127.0.0.1:49776 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49776 (request call id 1) could send its response
I20260619 16:49:32.514568 7808 ts_tablet_manager.cc:1905] T 086cb14238e245deb35d6390701ea2a8 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:32.515515 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:32.502869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35690 (local address 127.0.98.60:36871)
0619 16:49:32.502984 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:32.502987 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:32.503347 (+ 360us) server_negotiation.cc:407] Connection header received
0619 16:49:32.505206 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:32.505208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:32.505232 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:32.505285 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:32.506522 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.507341 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.508504 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.508834 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.508882 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:32.511183 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:32.511186 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:32.511186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:32.511197 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:32.513515 (+ 2318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:32.514428 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:32.514431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:32.514432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:32.514509 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:32.514751 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:32.514753 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:32.514754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:32.514923 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:32.515022 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:32.515279 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 16:49:32.515402 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 16:49:32.516594 7808 ts_tablet_manager.cc:1918] T 086cb14238e245deb35d6390701ea2a8 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:32.516683 7808 log.cc:1196] T 086cb14238e245deb35d6390701ea2a8 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-2/wal/wals/086cb14238e245deb35d6390701ea2a8
I20260619 16:49:32.516937 7808 ts_tablet_manager.cc:1939] T 086cb14238e245deb35d6390701ea2a8 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff: Deleting consensus metadata
W20260619 16:49:32.517552 7635 connection.cc:441] server connection from 127.0.0.1:56532 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56532 (request call id 1) could send its response
I20260619 16:49:32.518126 7806 tablet_replica.cc:330] T 086cb14238e245deb35d6390701ea2a8 P 830c2952c1c0432082d86c11093a1a1d: stopping tablet replica
I20260619 16:49:32.518337 7806 raft_consensus.cc:2229] T 086cb14238e245deb35d6390701ea2a8 P 830c2952c1c0432082d86c11093a1a1d [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:49:32.518512 7806 raft_consensus.cc:2258] T 086cb14238e245deb35d6390701ea2a8 P 830c2952c1c0432082d86c11093a1a1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:32.519173 7806 ts_tablet_manager.cc:1905] T 086cb14238e245deb35d6390701ea2a8 P 830c2952c1c0432082d86c11093a1a1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:32.521034 7806 ts_tablet_manager.cc:1918] T 086cb14238e245deb35d6390701ea2a8 P 830c2952c1c0432082d86c11093a1a1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:32.521119 7806 log.cc:1196] T 086cb14238e245deb35d6390701ea2a8 P 830c2952c1c0432082d86c11093a1a1d: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-0/wal/wals/086cb14238e245deb35d6390701ea2a8
I20260619 16:49:32.521387 7806 ts_tablet_manager.cc:1939] T 086cb14238e245deb35d6390701ea2a8 P 830c2952c1c0432082d86c11093a1a1d: Deleting consensus metadata
W20260619 16:49:32.521988 7364 connection.cc:441] server connection from 127.0.0.1:47988 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47988 (request call id 1) could send its response
I20260619 16:49:32.526993 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:32.517605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57908 (local address 127.0.98.61:44205)
0619 16:49:32.517677 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:32.517679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:32.517835 (+ 156us) server_negotiation.cc:407] Connection header received
0619 16:49:32.517968 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:32.517970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:32.517987 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:32.518064 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:32.519298 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.520115 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.521199 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.521554 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.521622 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:32.523994 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:32.523997 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:32.523997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:32.524007 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:32.525732 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:32.526133 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:32.526135 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:32.526135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:32.526227 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:32.526411 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:32.526412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:32.526412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:32.526566 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:32.526663 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:32.526779 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 16:49:32.526882 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 16:49:32.527105 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:32.517914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35700 (local address 127.0.98.60:36871)
0619 16:49:32.517965 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:32.517967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:32.518092 (+ 125us) server_negotiation.cc:407] Connection header received
0619 16:49:32.518297 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:32.518299 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:32.518310 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:32.518343 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:32.519298 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.520054 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.521170 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:32.521460 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:32.521510 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:32.523601 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:32.523603 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:32.523603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:32.523612 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:32.525666 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:32.526232 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:32.526233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:32.526234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:32.526291 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:32.526461 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:32.526462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:32.526463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:32.526625 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:32.526718 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:32.526904 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 16:49:32.527002 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 16:49:33.498243 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:33.497759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.98.3:47911)
0619 16:49:33.497971 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:33.498141 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":137,"threads_started":1}
I20260619 16:49:33.498541 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:33.498226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.98.1:36485)
0619 16:49:33.498400 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:33.498485 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":61,"threads_started":1}
W20260619 16:49:33.499048 7475 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36917 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
I20260619 16:49:33.499322 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:33.499001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.98.2:40871)
0619 16:49:33.499161 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:33.499227 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
W20260619 16:49:33.499384 7750 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36917 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
W20260619 16:49:33.499588 7612 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36917 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
I20260619 16:49:34.002780 7819 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae2a474c825646c2ab5e77aa28a5941a)
I20260619 16:49:34.002969 7819 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: false } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: false } }
I20260619 16:49:34.003334 7819 leader_election.cc:288] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae2a474c825646c2ab5e77aa28a5941a (127.0.98.62:36917), bfeed1e6da914f00952279d4b079cd75 (127.0.98.60:36871)
I20260619 16:49:34.003363 7820 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ae2a474c825646c2ab5e77aa28a5941a)
I20260619 16:49:34.003533 7820 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: false } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: false } }
I20260619 16:49:34.003832 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:34.003604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.0.1:54852)
0619 16:49:34.003721 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:34.003778 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
W20260619 16:49:34.003961 7087 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
I20260619 16:49:34.004122 7820 leader_election.cc:288] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae2a474c825646c2ab5e77aa28a5941a (127.0.98.62:36917), 4a5d873b1d944dbf99baa42af4f7b2f4 (127.0.98.61:44205)
I20260619 16:49:34.004191 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:34.003915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.0.1:54854)
0619 16:49:34.004113 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:34.004156 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":56,"threads_started":1}
W20260619 16:49:34.004278 7281 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
I20260619 16:49:34.004597 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:34.004505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.0.1:54866)
0619 16:49:34.004541 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:34.004567 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 16:49:34.004669 7087 leader_election.cc:334] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ae2a474c825646c2ab5e77aa28a5941a (127.0.98.62:36917): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
I20260619 16:49:34.004678 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:34.004597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.0.1:54870)
0619 16:49:34.004630 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:34.004653 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260619 16:49:34.004743 7281 leader_election.cc:334] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ae2a474c825646c2ab5e77aa28a5941a (127.0.98.62:36917): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
I20260619 16:49:34.008278 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:34.004057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57922 (local address 127.0.98.61:44205)
0619 16:49:34.004222 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:34.004223 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:34.004372 (+ 149us) server_negotiation.cc:407] Connection header received
0619 16:49:34.004402 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:34.004403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:34.004414 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:34.004484 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:34.004808 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:34.005347 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:34.007384 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:34.007806 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:34.007871 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:34.007888 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:34.008122 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 16:49:34.008180 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":38,"threads_started":1}
I20260619 16:49:34.008317 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:34.003715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35706 (local address 127.0.98.60:36871)
0619 16:49:34.003991 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:34.003994 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:34.004005 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:49:34.004041 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:34.004042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:34.004090 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:34.004524 (+ 434us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:34.004574 (+ 50us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:34.005286 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:34.007363 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:34.007894 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:34.007964 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:34.007979 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:34.008163 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 16:49:34.008220 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":245,"thread_start_us":63,"threads_started":1}
I20260619 16:49:34.008319 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:34.003937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44205 (local address 127.0.0.1:57922)
0619 16:49:34.004254 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:34.004263 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:34.004323 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:34.004532 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:34.004533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:34.004535 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:34.004618 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:34.004621 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:34.005681 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:34.005685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:34.007228 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:34.007234 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:34.007893 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:34.007894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:34.007965 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:34.007970 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:49:34.008093 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 16:49:34.008155 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":46,"threads_started":1}
I20260619 16:49:34.008277 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:34.003560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36871 (local address 127.0.0.1:35706)
0619 16:49:34.003719 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:34.003735 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:34.003806 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:34.004358 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:34.004359 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:34.004361 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:34.004447 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:34.004450 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:34.005366 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:34.005367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:34.007228 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:34.007234 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:34.007974 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:34.007975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:34.008035 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:34.008038 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:49:34.008092 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 16:49:34.008154 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":83,"threads_started":1}
I20260619 16:49:34.008679 7111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bfeed1e6da914f00952279d4b079cd75" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" is_pre_election: true
I20260619 16:49:34.008852 7111 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfeed1e6da914f00952279d4b079cd75 in term 1.
I20260619 16:49:34.008836 7305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bfeed1e6da914f00952279d4b079cd75" is_pre_election: true
I20260619 16:49:34.008955 7305 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a5d873b1d944dbf99baa42af4f7b2f4 in term 1.
I20260619 16:49:34.009032 7280 leader_election.cc:302] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a5d873b1d944dbf99baa42af4f7b2f4, bfeed1e6da914f00952279d4b079cd75; no voters: ae2a474c825646c2ab5e77aa28a5941a
I20260619 16:49:34.009145 7820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 16:49:34.009197 7820 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ae2a474c825646c2ab5e77aa28a5941a)
I20260619 16:49:34.009217 7820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:49:34.009256 7086 leader_election.cc:302] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a5d873b1d944dbf99baa42af4f7b2f4, bfeed1e6da914f00952279d4b079cd75; no voters: ae2a474c825646c2ab5e77aa28a5941a
I20260619 16:49:34.009361 7819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 16:49:34.009403 7819 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ae2a474c825646c2ab5e77aa28a5941a)
I20260619 16:49:34.009421 7819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:49:34.010387 7819 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: false } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: false } }
I20260619 16:49:34.010387 7820 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: false } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: false } }
I20260619 16:49:34.010556 7819 leader_election.cc:288] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [CANDIDATE]: Term 2 election: Requested vote from peers ae2a474c825646c2ab5e77aa28a5941a (127.0.98.62:36917), bfeed1e6da914f00952279d4b079cd75 (127.0.98.60:36871)
I20260619 16:49:34.010566 7820 leader_election.cc:288] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [CANDIDATE]: Term 2 election: Requested vote from peers ae2a474c825646c2ab5e77aa28a5941a (127.0.98.62:36917), 4a5d873b1d944dbf99baa42af4f7b2f4 (127.0.98.61:44205)
I20260619 16:49:34.010749 7111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bfeed1e6da914f00952279d4b079cd75" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4"
I20260619 16:49:34.010836 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:34.010723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.0.1:54890)
0619 16:49:34.010764 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:34.010800 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260619 16:49:34.010851 7111 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate bfeed1e6da914f00952279d4b079cd75 in current term 2: Already voted for candidate 4a5d873b1d944dbf99baa42af4f7b2f4 in this term.
I20260619 16:49:34.010901 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:34.010692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.0.1:54882)
0619 16:49:34.010846 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:34.010874 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260619 16:49:34.010885 7305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bfeed1e6da914f00952279d4b079cd75"
I20260619 16:49:34.010963 7305 raft_consensus.cc:2379] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a5d873b1d944dbf99baa42af4f7b2f4 in current term 2: Already voted for candidate bfeed1e6da914f00952279d4b079cd75 in this term.
I20260619 16:49:34.011143 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:34.011030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.0.1:54902)
0619 16:49:34.011069 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:34.011092 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 16:49:34.011246 7281 leader_election.cc:334] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ae2a474c825646c2ab5e77aa28a5941a (127.0.98.62:36917): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
I20260619 16:49:34.011288 7281 leader_election.cc:302] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bfeed1e6da914f00952279d4b079cd75; no voters: 4a5d873b1d944dbf99baa42af4f7b2f4, ae2a474c825646c2ab5e77aa28a5941a
I20260619 16:49:34.011381 7820 raft_consensus.cc:2726] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 16:49:34.011469 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:34.011353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.0.1:54916)
0619 16:49:34.011416 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:34.011446 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 16:49:34.011544 7087 leader_election.cc:334] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ae2a474c825646c2ab5e77aa28a5941a (127.0.98.62:36917): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
I20260619 16:49:34.011631 7087 leader_election.cc:302] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a5d873b1d944dbf99baa42af4f7b2f4; no voters: ae2a474c825646c2ab5e77aa28a5941a, bfeed1e6da914f00952279d4b079cd75
I20260619 16:49:34.011708 7819 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 16:49:34.499940 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:34.499672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.98.3:40261)
0619 16:49:34.499829 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:34.499873 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":78,"threads_started":1}
I20260619 16:49:34.499940 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:34.499622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.98.1:41241)
0619 16:49:34.499829 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:34.499873 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":109,"threads_started":1}
I20260619 16:49:34.500306 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:34.499867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.98.2:47579)
0619 16:49:34.500005 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:34.500254 (+ 249us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":75,"threads_started":1}
I20260619 16:49:35.500741 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:35.500462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.98.1:33011)
0619 16:49:35.500623 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:35.500671 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":88,"threads_started":1}
I20260619 16:49:35.500741 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:35.500457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.98.3:36843)
0619 16:49:35.500623 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:35.500670 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":88,"threads_started":1}
I20260619 16:49:35.500901 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:35.500675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.98.2:48023)
0619 16:49:35.500808 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:35.500847 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":66,"threads_started":1}
I20260619 16:49:35.919612 7835 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:35.919731 7835 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: false } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: false } }
I20260619 16:49:35.919940 7835 leader_election.cc:288] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ae2a474c825646c2ab5e77aa28a5941a (127.0.98.62:36917), 4a5d873b1d944dbf99baa42af4f7b2f4 (127.0.98.61:44205)
I20260619 16:49:35.920271 7111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bfeed1e6da914f00952279d4b079cd75" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" is_pre_election: true
I20260619 16:49:35.920377 7111 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfeed1e6da914f00952279d4b079cd75 in term 2.
I20260619 16:49:35.920476 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:35.920232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.0.1:55204)
0619 16:49:35.920371 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:35.920418 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20260619 16:49:35.920578 7280 leader_election.cc:302] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a5d873b1d944dbf99baa42af4f7b2f4, bfeed1e6da914f00952279d4b079cd75; no voters:
I20260619 16:49:35.920675 7835 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260619 16:49:35.920715 7835 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:49:35.920729 7835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [term 2 FOLLOWER]: Advancing to term 3
I20260619 16:49:35.920964 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:35.920871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.0.1:55208)
0619 16:49:35.920910 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:35.920938 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 16:49:35.921044 7281 leader_election.cc:334] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ae2a474c825646c2ab5e77aa28a5941a (127.0.98.62:36917): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
I20260619 16:49:35.921707 7835 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: false } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: false } }
I20260619 16:49:35.922108 7835 leader_election.cc:288] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [CANDIDATE]: Term 3 election: Requested vote from peers ae2a474c825646c2ab5e77aa28a5941a (127.0.98.62:36917), 4a5d873b1d944dbf99baa42af4f7b2f4 (127.0.98.61:44205)
I20260619 16:49:35.922364 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:35.922255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.0.1:55222)
0619 16:49:35.922296 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:35.922331 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260619 16:49:35.922391 7111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bfeed1e6da914f00952279d4b079cd75" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4"
I20260619 16:49:35.922506 7111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [term 2 FOLLOWER]: Advancing to term 3
I20260619 16:49:35.922847 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:35.922588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.0.1:55236)
0619 16:49:35.922785 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:35.922819 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177}
W20260619 16:49:35.922930 7281 leader_election.cc:334] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ae2a474c825646c2ab5e77aa28a5941a (127.0.98.62:36917): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
I20260619 16:49:35.924304 7111 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfeed1e6da914f00952279d4b079cd75 in term 3.
I20260619 16:49:35.924844 7280 leader_election.cc:302] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a5d873b1d944dbf99baa42af4f7b2f4, bfeed1e6da914f00952279d4b079cd75; no voters: ae2a474c825646c2ab5e77aa28a5941a
I20260619 16:49:35.925112 7835 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 16:49:35.925247 7835 raft_consensus.cc:686] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [term 3 LEADER]: Becoming Leader. State: Replica: bfeed1e6da914f00952279d4b079cd75, State: Running, Role: LEADER
I20260619 16:49:35.925534 7835 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: false } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: false } }
I20260619 16:49:35.926074 7838 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfeed1e6da914f00952279d4b079cd75. Latest consensus state: current_term: 3 leader_uuid: "bfeed1e6da914f00952279d4b079cd75" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: false } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: false } } }
I20260619 16:49:35.926225 7838 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:35.926433 7840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:35.927266 7840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:35.927387 7840 catalog_manager.cc:1259] Loaded cluster ID: d85fac49974d47e09e4548a836db6c9a
I20260619 16:49:35.927409 7840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:49:35.927809 7840 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:35.927908 7840 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75: Loaded TSK: 0
I20260619 16:49:35.928083 7840 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:49:35.949065 7295 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:49:35.951797 7295 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:49:35.953292 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:35.953158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.0.1:55260)
0619 16:49:35.953212 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:35.953250 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 16:49:35.953328 7111 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [term 3 FOLLOWER]: Refusing update from remote peer bfeed1e6da914f00952279d4b079cd75: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260619 16:49:35.953630 7835 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260619 16:49:35.953836 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:35.953705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.0.1:55276)
0619 16:49:35.953774 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:35.953809 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 16:49:35.953931 7281 consensus_peers.cc:487] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 -> Peer ae2a474c825646c2ab5e77aa28a5941a (127.0.98.62:36917): Couldn't send request to peer ae2a474c825646c2ab5e77aa28a5941a. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 16:49:35.954721 7842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfeed1e6da914f00952279d4b079cd75. Latest consensus state: current_term: 3 leader_uuid: "bfeed1e6da914f00952279d4b079cd75" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: false } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: false } } }
I20260619 16:49:35.954847 7842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:49:35.955384 7838 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "bfeed1e6da914f00952279d4b079cd75" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: false } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: false } } }
I20260619 16:49:35.955520 7842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "bfeed1e6da914f00952279d4b079cd75" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ae2a474c825646c2ab5e77aa28a5941a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36917 } } peers { permanent_uuid: "4a5d873b1d944dbf99baa42af4f7b2f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44205 } attrs { promote: false } } peers { permanent_uuid: "bfeed1e6da914f00952279d4b079cd75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36871 } attrs { promote: false } } }
I20260619 16:49:35.955580 7838 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:35.955622 7842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a5d873b1d944dbf99baa42af4f7b2f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:49:35.963723 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:35.958302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34327 (local address 127.0.0.1:49780)
0619 16:49:35.958366 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:35.958382 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:35.958459 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:35.959084 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:35.959086 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:35.959089 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:35.959178 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:35.959180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:35.961532 (+ 2352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:35.961533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:35.962595 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:35.962599 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:35.963396 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:35.963397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:35.963465 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:35.963472 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 16:49:35.963523 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 16:49:35.963606 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":3}
I20260619 16:49:35.963887 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:35.958879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48010 (local address 127.0.98.1:38953)
0619 16:49:35.959004 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:35.959006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:35.959018 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:49:35.959045 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:35.959045 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:35.959056 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:35.959293 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:35.959336 (+ 43us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:35.960047 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:35.962323 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:35.963250 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:35.963346 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:35.963365 (+ 19us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:35.963655 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 16:49:35.963794 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":69,"threads_started":1}
I20260619 16:49:35.964143 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:35.958500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38953 (local address 127.0.0.1:48010)
0619 16:49:35.958655 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:35.958678 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:35.958730 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:35.959163 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:35.959164 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:35.959166 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:35.959231 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:35.959233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:35.960245 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:35.960247 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:35.962163 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:35.962167 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:35.963377 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:35.963379 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:35.963462 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:35.963470 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 16:49:35.963526 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 16:49:35.963606 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":58,"thread_start_us":91,"threads_started":1}
I20260619 16:49:35.964447 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:35.958338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49780 (local address 127.0.98.2:34327)
0619 16:49:35.958936 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:35.958939 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:35.958948 (+ 9us) server_negotiation.cc:407] Connection header received
0619 16:49:35.958981 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:35.958982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:35.958995 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:35.959072 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:35.959242 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:35.960255 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:35.962729 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:35.963184 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:35.963264 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:35.963280 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:35.964286 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0619 16:49:35.964357 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":107,"threads_started":1}
I20260619 16:49:35.964439 7408 tablet_service.cc:1467] Processing CreateTablet for tablet c6d8204942d243aa8ec7f3fb67d8fa16 (DEFAULT_TABLE table=test-table [id=fd25fa88b2af42879eb275d0458674d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:35.964534 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:35.958445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38523 (local address 127.0.0.1:56548)
0619 16:49:35.958639 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:35.958650 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:35.958729 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:35.959162 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:35.959163 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:35.959165 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:35.959237 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:35.959238 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:35.960452 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:35.960454 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:35.962162 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:35.962167 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:35.964294 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:35.964295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:35.964359 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:35.964363 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:49:35.964411 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 16:49:35.964467 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":146,"thread_start_us":100,"threads_started":1}
I20260619 16:49:35.964567 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:35.958738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56548 (local address 127.0.98.3:38523)
0619 16:49:35.959006 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:35.959008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:35.959017 (+ 9us) server_negotiation.cc:407] Connection header received
0619 16:49:35.959044 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:35.959045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:35.959057 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:35.959154 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:35.959300 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:35.960291 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:35.963417 (+ 3126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:35.964168 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:35.964243 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:35.964258 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:35.964437 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 16:49:35.964494 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":78,"threads_started":1}
I20260619 16:49:35.964624 7408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d8204942d243aa8ec7f3fb67d8fa16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:35.964679 7543 tablet_service.cc:1467] Processing CreateTablet for tablet c6d8204942d243aa8ec7f3fb67d8fa16 (DEFAULT_TABLE table=test-table [id=fd25fa88b2af42879eb275d0458674d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:35.964797 7543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d8204942d243aa8ec7f3fb67d8fa16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:35.966085 7853 tablet_bootstrap.cc:492] T c6d8204942d243aa8ec7f3fb67d8fa16 P 6c278dee829942d6894a91b40527fba5: Bootstrap starting.
I20260619 16:49:35.966316 7854 tablet_bootstrap.cc:492] T c6d8204942d243aa8ec7f3fb67d8fa16 P 830c2952c1c0432082d86c11093a1a1d: Bootstrap starting.
I20260619 16:49:35.966619 7853 tablet_bootstrap.cc:654] T c6d8204942d243aa8ec7f3fb67d8fa16 P 6c278dee829942d6894a91b40527fba5: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:35.966735 7854 tablet_bootstrap.cc:654] T c6d8204942d243aa8ec7f3fb67d8fa16 P 830c2952c1c0432082d86c11093a1a1d: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:35.966763 7682 tablet_service.cc:1467] Processing CreateTablet for tablet c6d8204942d243aa8ec7f3fb67d8fa16 (DEFAULT_TABLE table=test-table [id=fd25fa88b2af42879eb275d0458674d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:35.966898 7682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d8204942d243aa8ec7f3fb67d8fa16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:35.967547 7854 tablet_bootstrap.cc:492] T c6d8204942d243aa8ec7f3fb67d8fa16 P 830c2952c1c0432082d86c11093a1a1d: No bootstrap required, opened a new log
I20260619 16:49:35.967645 7854 ts_tablet_manager.cc:1397] T c6d8204942d243aa8ec7f3fb67d8fa16 P 830c2952c1c0432082d86c11093a1a1d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:49:35.967829 7854 raft_consensus.cc:348] T c6d8204942d243aa8ec7f3fb67d8fa16 P 830c2952c1c0432082d86c11093a1a1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c278dee829942d6894a91b40527fba5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34327 } } peers { permanent_uuid: "830c2952c1c0432082d86c11093a1a1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38953 } } peers { permanent_uuid: "9dcaa1e6a3aa4e6ab19c3e19e70cdcff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38523 } }
I20260619 16:49:35.967917 7854 raft_consensus.cc:374] T c6d8204942d243aa8ec7f3fb67d8fa16 P 830c2952c1c0432082d86c11093a1a1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:35.967965 7853 tablet_bootstrap.cc:492] T c6d8204942d243aa8ec7f3fb67d8fa16 P 6c278dee829942d6894a91b40527fba5: No bootstrap required, opened a new log
I20260619 16:49:35.967937 7854 raft_consensus.cc:729] T c6d8204942d243aa8ec7f3fb67d8fa16 P 830c2952c1c0432082d86c11093a1a1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830c2952c1c0432082d86c11093a1a1d, State: Initialized, Role: FOLLOWER
I20260619 16:49:35.968024 7853 ts_tablet_manager.cc:1397] T c6d8204942d243aa8ec7f3fb67d8fa16 P 6c278dee829942d6894a91b40527fba5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:49:35.968035 7854 consensus_queue.cc:260] T c6d8204942d243aa8ec7f3fb67d8fa16 P 830c2952c1c0432082d86c11093a1a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c278dee829942d6894a91b40527fba5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34327 } } peers { permanent_uuid: "830c2952c1c0432082d86c11093a1a1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38953 } } peers { permanent_uuid: "9dcaa1e6a3aa4e6ab19c3e19e70cdcff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38523 } }
I20260619 16:49:35.968230 7854 ts_tablet_manager.cc:1428] T c6d8204942d243aa8ec7f3fb67d8fa16 P 830c2952c1c0432082d86c11093a1a1d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 16:49:35.968189 7853 raft_consensus.cc:348] T c6d8204942d243aa8ec7f3fb67d8fa16 P 6c278dee829942d6894a91b40527fba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c278dee829942d6894a91b40527fba5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34327 } } peers { permanent_uuid: "830c2952c1c0432082d86c11093a1a1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38953 } } peers { permanent_uuid: "9dcaa1e6a3aa4e6ab19c3e19e70cdcff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38523 } }
I20260619 16:49:35.968261 7853 raft_consensus.cc:374] T c6d8204942d243aa8ec7f3fb67d8fa16 P 6c278dee829942d6894a91b40527fba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:35.968279 7853 raft_consensus.cc:729] T c6d8204942d243aa8ec7f3fb67d8fa16 P 6c278dee829942d6894a91b40527fba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c278dee829942d6894a91b40527fba5, State: Initialized, Role: FOLLOWER
I20260619 16:49:35.968344 7853 consensus_queue.cc:260] T c6d8204942d243aa8ec7f3fb67d8fa16 P 6c278dee829942d6894a91b40527fba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c278dee829942d6894a91b40527fba5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34327 } } peers { permanent_uuid: "830c2952c1c0432082d86c11093a1a1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38953 } } peers { permanent_uuid: "9dcaa1e6a3aa4e6ab19c3e19e70cdcff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38523 } }
I20260619 16:49:35.968575 7853 ts_tablet_manager.cc:1428] T c6d8204942d243aa8ec7f3fb67d8fa16 P 6c278dee829942d6894a91b40527fba5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:35.968986 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:35.968630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.98.1:37573)
0619 16:49:35.968887 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:35.968945 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232}
I20260619 16:49:35.969296 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:35.969149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.98.2:43823)
0619 16:49:35.969228 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:35.969262 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 16:49:35.969496 7474 heartbeater.cc:503] Master 127.0.98.60:36871 was elected leader, sending a full tablet report...
I20260619 16:49:35.969564 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:35.969419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.98.1:49591)
0619 16:49:35.969470 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:35.969493 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 16:49:35.969846 7611 heartbeater.cc:503] Master 127.0.98.60:36871 was elected leader, sending a full tablet report...
I20260619 16:49:35.969921 7859 tablet_bootstrap.cc:492] T c6d8204942d243aa8ec7f3fb67d8fa16 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff: Bootstrap starting.
I20260619 16:49:35.970106 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:35.969959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.98.2:44211)
0619 16:49:35.970055 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:35.970081 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260619 16:49:35.970640 7859 tablet_bootstrap.cc:654] T c6d8204942d243aa8ec7f3fb67d8fa16 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:35.971520 7859 tablet_bootstrap.cc:492] T c6d8204942d243aa8ec7f3fb67d8fa16 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff: No bootstrap required, opened a new log
I20260619 16:49:35.971607 7859 ts_tablet_manager.cc:1397] T c6d8204942d243aa8ec7f3fb67d8fa16 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 16:49:35.971755 7859 raft_consensus.cc:348] T c6d8204942d243aa8ec7f3fb67d8fa16 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c278dee829942d6894a91b40527fba5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34327 } } peers { permanent_uuid: "830c2952c1c0432082d86c11093a1a1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38953 } } peers { permanent_uuid: "9dcaa1e6a3aa4e6ab19c3e19e70cdcff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38523 } }
I20260619 16:49:35.971827 7859 raft_consensus.cc:374] T c6d8204942d243aa8ec7f3fb67d8fa16 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:35.971840 7859 raft_consensus.cc:729] T c6d8204942d243aa8ec7f3fb67d8fa16 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dcaa1e6a3aa4e6ab19c3e19e70cdcff, State: Initialized, Role: FOLLOWER
I20260619 16:49:35.971879 7859 consensus_queue.cc:260] T c6d8204942d243aa8ec7f3fb67d8fa16 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c278dee829942d6894a91b40527fba5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34327 } } peers { permanent_uuid: "830c2952c1c0432082d86c11093a1a1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38953 } } peers { permanent_uuid: "9dcaa1e6a3aa4e6ab19c3e19e70cdcff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38523 } }
I20260619 16:49:35.972052 7859 ts_tablet_manager.cc:1428] T c6d8204942d243aa8ec7f3fb67d8fa16 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260619 16:49:35.972427 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:35.972316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.98.3:40459)
0619 16:49:35.972358 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:35.972394 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260619 16:49:36.003242 7858 raft_consensus.cc:482] T c6d8204942d243aa8ec7f3fb67d8fa16 P 6c278dee829942d6894a91b40527fba5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:36.003372 7858 raft_consensus.cc:504] T c6d8204942d243aa8ec7f3fb67d8fa16 P 6c278dee829942d6894a91b40527fba5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c278dee829942d6894a91b40527fba5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34327 } } peers { permanent_uuid: "830c2952c1c0432082d86c11093a1a1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38953 } } peers { permanent_uuid: "9dcaa1e6a3aa4e6ab19c3e19e70cdcff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38523 } }
I20260619 16:49:36.003750 7858 leader_election.cc:288] T c6d8204942d243aa8ec7f3fb67d8fa16 P 6c278dee829942d6894a91b40527fba5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 830c2952c1c0432082d86c11093a1a1d (127.0.98.1:38953), 9dcaa1e6a3aa4e6ab19c3e19e70cdcff (127.0.98.3:38523)
I20260619 16:49:36.008226 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:36.004073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38523 (local address 127.0.98.2:48697)
0619 16:49:36.004148 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:36.004163 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:36.004238 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:36.004751 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:36.004753 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:36.004758 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:36.004875 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:36.004878 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:36.005703 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:36.005704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:36.007200 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:36.007203 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:36.007934 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:36.007935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:36.007999 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:36.008005 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 16:49:36.008067 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 16:49:36.008114 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":72}
I20260619 16:49:36.008219 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:36.004406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48697 (local address 127.0.98.3:38523)
0619 16:49:36.004511 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:36.004513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:36.004523 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:49:36.004594 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:36.004596 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:36.004612 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:36.004716 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:36.004985 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:36.005569 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:36.007314 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:36.007847 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:36.007911 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:36.007925 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:36.008084 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 16:49:36.008135 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 16:49:36.008359 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:36.004073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38953 (local address 127.0.98.2:45269)
0619 16:49:36.004789 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:36.004800 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:36.004868 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:36.004992 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:36.004993 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:36.004995 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:36.005059 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:36.005060 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:36.006281 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:36.006283 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:36.007206 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:36.007209 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:36.007997 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:36.007998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:36.008165 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:36.008169 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:49:36.008231 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 16:49:36.008276 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"mutex_wait_us":306,"thread_start_us":301,"threads_started":1}
I20260619 16:49:36.008383 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:36.004208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45269 (local address 127.0.98.1:38953)
0619 16:49:36.004337 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:36.004339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:36.004873 (+ 534us) server_negotiation.cc:407] Connection header received
0619 16:49:36.004912 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:36.004914 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:36.004929 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:36.004983 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:36.005221 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:36.006088 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:36.007370 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:36.007915 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:36.007989 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:36.008004 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:36.008254 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 16:49:36.008313 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260619 16:49:36.008617 7428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d8204942d243aa8ec7f3fb67d8fa16" candidate_uuid: "6c278dee829942d6894a91b40527fba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830c2952c1c0432082d86c11093a1a1d" is_pre_election: true
I20260619 16:49:36.008869 7428 raft_consensus.cc:2454] T c6d8204942d243aa8ec7f3fb67d8fa16 P 830c2952c1c0432082d86c11093a1a1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c278dee829942d6894a91b40527fba5 in term 0.
I20260619 16:49:36.008688 7703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d8204942d243aa8ec7f3fb67d8fa16" candidate_uuid: "6c278dee829942d6894a91b40527fba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dcaa1e6a3aa4e6ab19c3e19e70cdcff" is_pre_election: true
I20260619 16:49:36.009102 7703 raft_consensus.cc:2454] T c6d8204942d243aa8ec7f3fb67d8fa16 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c278dee829942d6894a91b40527fba5 in term 0.
I20260619 16:49:36.009106 7500 leader_election.cc:302] T c6d8204942d243aa8ec7f3fb67d8fa16 P 6c278dee829942d6894a91b40527fba5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c278dee829942d6894a91b40527fba5, 830c2952c1c0432082d86c11093a1a1d; no voters:
I20260619 16:49:36.009285 7858 raft_consensus.cc:2781] T c6d8204942d243aa8ec7f3fb67d8fa16 P 6c278dee829942d6894a91b40527fba5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:49:36.009336 7858 raft_consensus.cc:482] T c6d8204942d243aa8ec7f3fb67d8fa16 P 6c278dee829942d6894a91b40527fba5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:49:36.009358 7858 raft_consensus.cc:3037] T c6d8204942d243aa8ec7f3fb67d8fa16 P 6c278dee829942d6894a91b40527fba5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:36.010308 7858 raft_consensus.cc:504] T c6d8204942d243aa8ec7f3fb67d8fa16 P 6c278dee829942d6894a91b40527fba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c278dee829942d6894a91b40527fba5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34327 } } peers { permanent_uuid: "830c2952c1c0432082d86c11093a1a1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38953 } } peers { permanent_uuid: "9dcaa1e6a3aa4e6ab19c3e19e70cdcff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38523 } }
I20260619 16:49:36.010475 7858 leader_election.cc:288] T c6d8204942d243aa8ec7f3fb67d8fa16 P 6c278dee829942d6894a91b40527fba5 [CANDIDATE]: Term 1 election: Requested vote from peers 830c2952c1c0432082d86c11093a1a1d (127.0.98.1:38953), 9dcaa1e6a3aa4e6ab19c3e19e70cdcff (127.0.98.3:38523)
I20260619 16:49:36.010665 7428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d8204942d243aa8ec7f3fb67d8fa16" candidate_uuid: "6c278dee829942d6894a91b40527fba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830c2952c1c0432082d86c11093a1a1d"
I20260619 16:49:36.010704 7703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6d8204942d243aa8ec7f3fb67d8fa16" candidate_uuid: "6c278dee829942d6894a91b40527fba5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dcaa1e6a3aa4e6ab19c3e19e70cdcff"
I20260619 16:49:36.010764 7428 raft_consensus.cc:3037] T c6d8204942d243aa8ec7f3fb67d8fa16 P 830c2952c1c0432082d86c11093a1a1d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:36.010782 7703 raft_consensus.cc:3037] T c6d8204942d243aa8ec7f3fb67d8fa16 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:36.011715 7428 raft_consensus.cc:2454] T c6d8204942d243aa8ec7f3fb67d8fa16 P 830c2952c1c0432082d86c11093a1a1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c278dee829942d6894a91b40527fba5 in term 1.
I20260619 16:49:36.011713 7703 raft_consensus.cc:2454] T c6d8204942d243aa8ec7f3fb67d8fa16 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c278dee829942d6894a91b40527fba5 in term 1.
I20260619 16:49:36.011974 7499 leader_election.cc:302] T c6d8204942d243aa8ec7f3fb67d8fa16 P 6c278dee829942d6894a91b40527fba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c278dee829942d6894a91b40527fba5, 9dcaa1e6a3aa4e6ab19c3e19e70cdcff; no voters:
I20260619 16:49:36.012097 7858 raft_consensus.cc:2781] T c6d8204942d243aa8ec7f3fb67d8fa16 P 6c278dee829942d6894a91b40527fba5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:36.012293 7858 raft_consensus.cc:686] T c6d8204942d243aa8ec7f3fb67d8fa16 P 6c278dee829942d6894a91b40527fba5 [term 1 LEADER]: Becoming Leader. State: Replica: 6c278dee829942d6894a91b40527fba5, State: Running, Role: LEADER
I20260619 16:49:36.012374 7858 consensus_queue.cc:237] T c6d8204942d243aa8ec7f3fb67d8fa16 P 6c278dee829942d6894a91b40527fba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c278dee829942d6894a91b40527fba5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34327 } } peers { permanent_uuid: "830c2952c1c0432082d86c11093a1a1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38953 } } peers { permanent_uuid: "9dcaa1e6a3aa4e6ab19c3e19e70cdcff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38523 } }
I20260619 16:49:36.012681 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:36.012552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.98.2:41963)
0619 16:49:36.012598 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:36.012644 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260619 16:49:36.013346 7294 catalog_manager.cc:5495] T c6d8204942d243aa8ec7f3fb67d8fa16 P 6c278dee829942d6894a91b40527fba5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c278dee829942d6894a91b40527fba5 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6c278dee829942d6894a91b40527fba5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c278dee829942d6894a91b40527fba5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34327 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "830c2952c1c0432082d86c11093a1a1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dcaa1e6a3aa4e6ab19c3e19e70cdcff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38523 } health_report { overall_health: UNKNOWN } } }
I20260619 16:49:36.029227 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:36.025944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49794 (local address 127.0.98.2:34327)
0619 16:49:36.026008 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:36.026010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:36.026022 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:49:36.026339 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:36.026341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:36.026375 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:36.026462 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:36.026614 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:36.027144 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:36.028165 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:36.028389 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:36.028454 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:36.028693 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:36.028866 (+ 173us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:36.028972 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:36.029083 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 16:49:36.029120 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 16:49:36.030972 7703 raft_consensus.cc:1261] T c6d8204942d243aa8ec7f3fb67d8fa16 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff [term 1 FOLLOWER]: Refusing update from remote peer 6c278dee829942d6894a91b40527fba5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:36.031183 7428 raft_consensus.cc:1261] T c6d8204942d243aa8ec7f3fb67d8fa16 P 830c2952c1c0432082d86c11093a1a1d [term 1 FOLLOWER]: Refusing update from remote peer 6c278dee829942d6894a91b40527fba5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:36.031373 7858 consensus_queue.cc:1035] T c6d8204942d243aa8ec7f3fb67d8fa16 P 6c278dee829942d6894a91b40527fba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dcaa1e6a3aa4e6ab19c3e19e70cdcff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:36.031596 7863 consensus_queue.cc:1035] T c6d8204942d243aa8ec7f3fb67d8fa16 P 6c278dee829942d6894a91b40527fba5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "830c2952c1c0432082d86c11093a1a1d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:36.031951 7749 heartbeater.cc:503] Master 127.0.98.60:36871 was elected leader, sending a full tablet report...
I20260619 16:49:36.032546 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:36.032378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.98.1:46693)
0619 16:49:36.032447 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:36.032499 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 16:49:36.032851 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:36.032684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.98.3:44663)
0619 16:49:36.032785 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:36.032818 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260619 16:49:36.034114 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:36.033986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.98.2:37063)
0619 16:49:36.034027 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:36.034068 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260619 16:49:36.034509 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:36.034399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.98.3:33259)
0619 16:49:36.034441 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:36.034474 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 16:49:36.035109 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:36.034998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.98.1:49381)
0619 16:49:36.035044 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:36.035071 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 16:49:36.036152 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:36.036029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36917 (local address 127.0.98.2:34027)
0619 16:49:36.036076 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:36.036112 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 16:49:36.038254 7295 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:36.038416 7295 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:36.041457 7295 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bfeed1e6da914f00952279d4b079cd75: Sending DeleteTablet for 3 replicas of tablet c6d8204942d243aa8ec7f3fb67d8fa16
I20260619 16:49:36.042089 7543 tablet_service.cc:1514] Processing DeleteTablet for tablet c6d8204942d243aa8ec7f3fb67d8fa16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:36 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:49780
I20260619 16:49:36.042156 7408 tablet_service.cc:1514] Processing DeleteTablet for tablet c6d8204942d243aa8ec7f3fb67d8fa16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:36 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:48010
I20260619 16:49:36.042568 7682 tablet_service.cc:1514] Processing DeleteTablet for tablet c6d8204942d243aa8ec7f3fb67d8fa16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:36 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:56548
I20260619 16:49:36.042609 7877 tablet_replica.cc:330] T c6d8204942d243aa8ec7f3fb67d8fa16 P 830c2952c1c0432082d86c11093a1a1d: stopping tablet replica
I20260619 16:49:36.042819 7878 tablet_replica.cc:330] T c6d8204942d243aa8ec7f3fb67d8fa16 P 6c278dee829942d6894a91b40527fba5: stopping tablet replica
I20260619 16:49:36.042899 7878 raft_consensus.cc:2229] T c6d8204942d243aa8ec7f3fb67d8fa16 P 6c278dee829942d6894a91b40527fba5 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:49:36.043036 7878 raft_consensus.cc:2258] T c6d8204942d243aa8ec7f3fb67d8fa16 P 6c278dee829942d6894a91b40527fba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:36.043028 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 7249
I20260619 16:49:36.044459 7879 tablet_replica.cc:330] T c6d8204942d243aa8ec7f3fb67d8fa16 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff: stopping tablet replica
I20260619 16:49:36.044571 7879 raft_consensus.cc:2229] T c6d8204942d243aa8ec7f3fb67d8fa16 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:36.044643 7879 raft_consensus.cc:2258] T c6d8204942d243aa8ec7f3fb67d8fa16 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:36.045176 7879 ts_tablet_manager.cc:1905] T c6d8204942d243aa8ec7f3fb67d8fa16 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:36.045184 7878 ts_tablet_manager.cc:1905] T c6d8204942d243aa8ec7f3fb67d8fa16 P 6c278dee829942d6894a91b40527fba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:36.047513 7879 ts_tablet_manager.cc:1918] T c6d8204942d243aa8ec7f3fb67d8fa16 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:36.047580 7878 ts_tablet_manager.cc:1918] T c6d8204942d243aa8ec7f3fb67d8fa16 P 6c278dee829942d6894a91b40527fba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:36.047645 7878 log.cc:1196] T c6d8204942d243aa8ec7f3fb67d8fa16 P 6c278dee829942d6894a91b40527fba5: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-1/wal/wals/c6d8204942d243aa8ec7f3fb67d8fa16
I20260619 16:49:36.047821 7879 log.cc:1196] T c6d8204942d243aa8ec7f3fb67d8fa16 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781887741903642-392-0/minicluster-data/ts-2/wal/wals/c6d8204942d243aa8ec7f3fb67d8fa16
I20260619 16:49:36.047883 7878 ts_tablet_manager.cc:1939] T c6d8204942d243aa8ec7f3fb67d8fa16 P 6c278dee829942d6894a91b40527fba5: Deleting consensus metadata
I20260619 16:49:36.048197 7879 ts_tablet_manager.cc:1939] T c6d8204942d243aa8ec7f3fb67d8fa16 P 9dcaa1e6a3aa4e6ab19c3e19e70cdcff: Deleting consensus metadata
I20260619 16:49:36.048597 7280 catalog_manager.cc:4841] TS 6c278dee829942d6894a91b40527fba5 (127.0.98.2:34327): tablet c6d8204942d243aa8ec7f3fb67d8fa16 (table test-table [id=fd25fa88b2af42879eb275d0458674d6]) successfully deleted
I20260619 16:49:36.048740 7280 catalog_manager.cc:4841] TS 9dcaa1e6a3aa4e6ab19c3e19e70cdcff (127.0.98.3:38523): tablet c6d8204942d243aa8ec7f3fb67d8fa16 (table test-table [id=fd25fa88b2af42879eb275d0458674d6]) successfully deleted
W20260619 16:49:36.051635 7849 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:36.051333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38953 (local address 127.0.0.1:48020)
0619 16:49:36.051406 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:36.051420 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:36.051479 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:36.051560 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38953: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 16:49:36.051923 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 7482
W20260619 16:49:36.051950 7281 catalog_manager.cc:4584] TS 830c2952c1c0432082d86c11093a1a1d (127.0.98.1:38953): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c6d8204942d243aa8ec7f3fb67d8fa16: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38953: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260619 16:49:36.057777 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 7618
I20260619 16:49:36.069300 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 6967
I20260619 16:49:36.074333 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 7155
2026-06-19T16:49:36Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7187 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:49:36 dist-test-slave-jsw9 krb5kdc[7882](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:49:36 dist-test-slave-jsw9 krb5kdc[7882](info): set up 2 sockets
Jun 19 16:49:36 dist-test-slave-jsw9 krb5kdc[7882](info): commencing operation
krb5kdc: starting...
W20260619 16:49:37.108304 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:49:37 dist-test-slave-jsw9 krb5kdc[7882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:49:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:49:37Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260619 16:49:37.277604 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37881
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:37.399505 7895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:37.400221 7895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:37.400293 7895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:37.402549 7895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:37.402644 7895 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 16:49:37.402660 7895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:37.402674 7895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:37.402685 7895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:37.402697 7895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:49:37.405028 7895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37881
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:37.405416 7895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:37.405722 7895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:37.412415 7900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:37.412447 7895 server_base.cc:1034] running on GCE node
W20260619 16:49:37.412407 7901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:37.412794 7903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:37.413121 7895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:37.413470 7895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:37.414659 7895 hybrid_clock.cc:648] HybridClock initialized: now 1781887777414638 us; error 34 us; skew 500 ppm
Jun 19 16:49:37 dist-test-slave-jsw9 krb5kdc[7882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887777, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:37.417896 7895 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260619 16:49:37.418546 7895 webserver.cc:460] Webserver started at http://127.0.98.62:44211/ using document root <none> and password file <none>
I20260619 16:49:37.418850 7895 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:37.418915 7895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:37.419080 7895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:37.420511 7895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "384f080274ce451a8180fbc2fbda74ec"
format_stamp: "Formatted at 2026-06-19 16:49:37 on dist-test-slave-jsw9"
server_key: "9cb1b73761a816521106bd9c34253b91"
server_key_iv: "19f74285792888735ce744c5ae6313ec"
server_key_version: "encryptionkey@0"
I20260619 16:49:37.421011 7895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "384f080274ce451a8180fbc2fbda74ec"
format_stamp: "Formatted at 2026-06-19 16:49:37 on dist-test-slave-jsw9"
server_key: "9cb1b73761a816521106bd9c34253b91"
server_key_iv: "19f74285792888735ce744c5ae6313ec"
server_key_version: "encryptionkey@0"
I20260619 16:49:37.422873 7895 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260619 16:49:37.424196 7910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:37.424491 7895 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:37.424618 7895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "384f080274ce451a8180fbc2fbda74ec"
format_stamp: "Formatted at 2026-06-19 16:49:37 on dist-test-slave-jsw9"
server_key: "9cb1b73761a816521106bd9c34253b91"
server_key_iv: "19f74285792888735ce744c5ae6313ec"
server_key_version: "encryptionkey@0"
I20260619 16:49:37.424711 7895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:37.441757 7895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:37.444540 7895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:37.444759 7895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:37.450572 7895 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37881
I20260619 16:49:37.450614 7972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37881 every 8 connection(s)
I20260619 16:49:37.451190 7895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:37.452018 7973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:37.455860 7973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384f080274ce451a8180fbc2fbda74ec: Bootstrap starting.
I20260619 16:49:37.456085 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 7895
I20260619 16:49:37.456187 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:49:37.456429 392 external_mini_cluster.cc:1428] Setting key b69b9d1d4b823c783b2c97b61e0f11bb
I20260619 16:49:37.456840 7973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 384f080274ce451a8180fbc2fbda74ec: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:37.457240 7973 log.cc:824] T 00000000000000000000000000000000 P 384f080274ce451a8180fbc2fbda74ec: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:37.458945 7973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384f080274ce451a8180fbc2fbda74ec: No bootstrap required, opened a new log
I20260619 16:49:37.461385 7973 raft_consensus.cc:348] T 00000000000000000000000000000000 P 384f080274ce451a8180fbc2fbda74ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384f080274ce451a8180fbc2fbda74ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37881 } }
I20260619 16:49:37.461614 7973 raft_consensus.cc:374] T 00000000000000000000000000000000 P 384f080274ce451a8180fbc2fbda74ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:37.461644 7973 raft_consensus.cc:729] T 00000000000000000000000000000000 P 384f080274ce451a8180fbc2fbda74ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 384f080274ce451a8180fbc2fbda74ec, State: Initialized, Role: FOLLOWER
I20260619 16:49:37.461793 7973 consensus_queue.cc:260] T 00000000000000000000000000000000 P 384f080274ce451a8180fbc2fbda74ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384f080274ce451a8180fbc2fbda74ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37881 } }
I20260619 16:49:37.461874 7973 raft_consensus.cc:388] T 00000000000000000000000000000000 P 384f080274ce451a8180fbc2fbda74ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:37.461906 7973 raft_consensus.cc:482] T 00000000000000000000000000000000 P 384f080274ce451a8180fbc2fbda74ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:37.461956 7973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 384f080274ce451a8180fbc2fbda74ec [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:37.462795 7973 raft_consensus.cc:504] T 00000000000000000000000000000000 P 384f080274ce451a8180fbc2fbda74ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384f080274ce451a8180fbc2fbda74ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37881 } }
I20260619 16:49:37.462908 7973 leader_election.cc:302] T 00000000000000000000000000000000 P 384f080274ce451a8180fbc2fbda74ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 384f080274ce451a8180fbc2fbda74ec; no voters:
I20260619 16:49:37.463186 7973 leader_election.cc:288] T 00000000000000000000000000000000 P 384f080274ce451a8180fbc2fbda74ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:49:37.463238 7978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 384f080274ce451a8180fbc2fbda74ec [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:37.463377 7978 raft_consensus.cc:686] T 00000000000000000000000000000000 P 384f080274ce451a8180fbc2fbda74ec [term 1 LEADER]: Becoming Leader. State: Replica: 384f080274ce451a8180fbc2fbda74ec, State: Running, Role: LEADER
I20260619 16:49:37.463478 7978 consensus_queue.cc:237] T 00000000000000000000000000000000 P 384f080274ce451a8180fbc2fbda74ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384f080274ce451a8180fbc2fbda74ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37881 } }
I20260619 16:49:37.463569 7973 sys_catalog.cc:564] T 00000000000000000000000000000000 P 384f080274ce451a8180fbc2fbda74ec [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:37.463886 7978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 384f080274ce451a8180fbc2fbda74ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 384f080274ce451a8180fbc2fbda74ec. Latest consensus state: current_term: 1 leader_uuid: "384f080274ce451a8180fbc2fbda74ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384f080274ce451a8180fbc2fbda74ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37881 } } }
I20260619 16:49:37.463976 7978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 384f080274ce451a8180fbc2fbda74ec [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:37.464147 7979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 384f080274ce451a8180fbc2fbda74ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "384f080274ce451a8180fbc2fbda74ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384f080274ce451a8180fbc2fbda74ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37881 } } }
I20260619 16:49:37.464216 7979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 384f080274ce451a8180fbc2fbda74ec [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:37.464351 7983 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:37.465000 7983 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:37.467067 7983 catalog_manager.cc:1347] Generated new cluster ID: d4f3f9e76ced4849b52d09d75ac0ac7f
I20260619 16:49:37.467139 7983 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 16:49:37 dist-test-slave-jsw9 krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 16:49:37.474362 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:37.457281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54448 (local address 127.0.98.62:37881)
0619 16:49:37.457539 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:37.457550 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:37.457618 (+ 68us) server_negotiation.cc:407] Connection header received
0619 16:49:37.459104 (+ 1486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:37.459115 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:37.459197 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:37.459596 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:37.460808 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.462836 (+ 2028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.465156 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.465844 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.465948 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:37.469382 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:37.469389 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:37.469390 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:37.469410 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:37.471834 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:37.472436 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:37.472440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:37.472440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:37.472585 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:37.472824 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:37.472824 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:37.472825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:37.473345 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:37.473576 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:37.473726 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 16:49:37.474011 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":108,"threads_started":1}
I20260619 16:49:37.481393 7983 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:49:37.482071 7983 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:37.487463 7983 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 384f080274ce451a8180fbc2fbda74ec: Generated new TSK 0
I20260619 16:49:37.487874 7983 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260619 16:49:37.527786 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37881
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:37.640749 7999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:37.641091 7999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:37.641127 7999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:37.643023 7999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:37.643121 7999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:37.643151 7999 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:49:37.643172 7999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:49:37.645471 7999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:37.645804 7999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:37.646133 7999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:37.654035 8007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:37.655177 8005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:37.655184 8004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:37.655232 7999 server_base.cc:1034] running on GCE node
I20260619 16:49:37.655891 7999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:37.656265 7999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:37.657573 7999 hybrid_clock.cc:648] HybridClock initialized: now 1781887777657537 us; error 40 us; skew 500 ppm
Jun 19 16:49:37 dist-test-slave-jsw9 krb5kdc[7882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887777, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:37.661717 7999 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 16:49:37.662573 7999 webserver.cc:460] Webserver started at http://127.0.98.1:35653/ using document root <none> and password file <none>
I20260619 16:49:37.663090 7999 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:37.663306 7999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:37.663589 7999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:37.665446 7999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-0/data/instance:
uuid: "edba86e2b7a74ab880b1f716db2c7471"
format_stamp: "Formatted at 2026-06-19 16:49:37 on dist-test-slave-jsw9"
server_key: "f605d3f16dcab3155eefc30dee823d1d"
server_key_iv: "662723df8277f0995d02338f45761f88"
server_key_version: "encryptionkey@0"
I20260619 16:49:37.666107 7999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-0/wal/instance:
uuid: "edba86e2b7a74ab880b1f716db2c7471"
format_stamp: "Formatted at 2026-06-19 16:49:37 on dist-test-slave-jsw9"
server_key: "f605d3f16dcab3155eefc30dee823d1d"
server_key_iv: "662723df8277f0995d02338f45761f88"
server_key_version: "encryptionkey@0"
I20260619 16:49:37.667834 7999 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:49:37.669298 8014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:37.669721 7999 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:37.669847 7999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-0/wal
uuid: "edba86e2b7a74ab880b1f716db2c7471"
format_stamp: "Formatted at 2026-06-19 16:49:37 on dist-test-slave-jsw9"
server_key: "f605d3f16dcab3155eefc30dee823d1d"
server_key_iv: "662723df8277f0995d02338f45761f88"
server_key_version: "encryptionkey@0"
I20260619 16:49:37.669924 7999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:37.688650 7999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:37.691496 7999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:37.691641 7999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:37.692351 7999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:37.692400 7999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:37.692442 7999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:37.692462 7999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:37.700682 7999 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37025
I20260619 16:49:37.700867 8129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37025 every 8 connection(s)
I20260619 16:49:37.701201 7999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 16:49:37 dist-test-slave-jsw9 krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887777, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 16:49:37.705960 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 7999
I20260619 16:49:37.706090 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:49:37.706393 392 external_mini_cluster.cc:1428] Setting key dc2ff9db47e0993f74c5e927c4a81737
I20260619 16:49:37.707093 8023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:37.693108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37881 (local address 127.0.98.1:40873)
0619 16:49:37.693304 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:37.693348 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:37.694150 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:37.694523 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:37.694529 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:37.694959 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:37.695293 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:37.695302 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.696492 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.696493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:37.698095 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:37.698101 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.698839 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.698840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:37.699053 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:37.699887 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:37.699904 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:37.701854 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:37.704541 (+ 2687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:37.704560 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:37.704564 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:37.705179 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:37.705462 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:37.705463 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:37.705463 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:37.705665 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:37.706063 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:37.706076 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:37.706355 (+ 279us) client_negotiation.cc:769] Sending connection context
0619 16:49:37.706502 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 16:49:37.706773 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":99,"threads_started":1}
I20260619 16:49:37.707679 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:37.693126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40873 (local address 127.0.98.62:37881)
0619 16:49:37.693190 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:37.693192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:37.693545 (+ 353us) server_negotiation.cc:407] Connection header received
0619 16:49:37.694315 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:37.694317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:37.694335 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:37.694409 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:37.695488 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.696341 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.698343 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.698701 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.698806 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:37.701997 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:37.701999 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:37.702000 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:37.702010 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:37.704268 (+ 2258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:37.705276 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:37.705278 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:37.705279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:37.705372 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:37.705773 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:37.705775 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:37.705776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:37.705978 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:37.706057 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:37.707434 (+ 1377us) server_negotiation.cc:299] Negotiation successful
0619 16:49:37.707573 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 16:49:37.708856 8130 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37881
I20260619 16:49:37.709045 8130 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:37.709368 8130 heartbeater.cc:511] Master 127.0.98.62:37881 requested a full tablet report, sending...
I20260619 16:49:37.710479 7927 ts_manager.cc:194] Registered new tserver with Master: edba86e2b7a74ab880b1f716db2c7471 (127.0.98.1:37025)
I20260619 16:49:37.711479 7927 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40873
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 16:49:37.719866 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:37.711755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54151 (local address 127.0.98.62:37881)
0619 16:49:37.711849 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:37.711851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:37.711863 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:49:37.711900 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:37.711901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:37.711914 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:37.711971 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:37.712487 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.713051 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.714133 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.714419 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.714487 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:37.716207 (+ 1720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:37.716209 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:37.716209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:37.716221 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:37.718309 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:37.718927 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:37.718929 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:37.718930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:37.718999 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:37.719261 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:37.719262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:37.719263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:37.719413 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:37.719477 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:37.719703 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 16:49:37.719783 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:49:37.719880 8137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:37.710276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37881 (local address 127.0.98.1:54151)
0619 16:49:37.710440 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:37.710458 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:37.710518 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:37.712043 (+ 1525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:37.712045 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:37.712313 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:37.712393 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:37.712397 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.713174 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.713176 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:37.714009 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:37.714014 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.714500 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.714501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:37.714583 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:37.715467 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:37.715477 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:37.716019 (+ 542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:37.718427 (+ 2408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:37.718429 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:37.718430 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:37.718787 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:37.719080 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:37.719081 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:37.719082 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:37.719163 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:37.719498 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:37.719500 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:37.719564 (+ 64us) client_negotiation.cc:769] Sending connection context
0619 16:49:37.719635 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 16:49:37.719779 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":86,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260619 16:49:37.746604 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37881
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:37.848148 8139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:37.848495 8139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:37.848541 8139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:37.850314 8139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:37.850404 8139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:37.850440 8139 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:49:37.850468 8139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:49:37.852594 8139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:37.852944 8139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:37.853238 8139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:37.860436 8147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:37.860599 8139 server_base.cc:1034] running on GCE node
W20260619 16:49:37.860460 8144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:37.860786 8145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:37.861068 8139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:37.861413 8139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:37.862593 8139 hybrid_clock.cc:648] HybridClock initialized: now 1781887777862555 us; error 52 us; skew 500 ppm
Jun 19 16:49:37 dist-test-slave-jsw9 krb5kdc[7882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887777, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:37.866338 8139 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 16:49:37.866993 8139 webserver.cc:460] Webserver started at http://127.0.98.2:43075/ using document root <none> and password file <none>
I20260619 16:49:37.867254 8139 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:37.867313 8139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:37.867462 8139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:37.868861 8139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-1/data/instance:
uuid: "0c052d73c57e481eaa3699c11d96116d"
format_stamp: "Formatted at 2026-06-19 16:49:37 on dist-test-slave-jsw9"
server_key: "86c51f324c28cf37adf281529e3c483e"
server_key_iv: "c615aafdf977ab70ee766f3fa7674746"
server_key_version: "encryptionkey@0"
I20260619 16:49:37.869325 8139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0c052d73c57e481eaa3699c11d96116d"
format_stamp: "Formatted at 2026-06-19 16:49:37 on dist-test-slave-jsw9"
server_key: "86c51f324c28cf37adf281529e3c483e"
server_key_iv: "c615aafdf977ab70ee766f3fa7674746"
server_key_version: "encryptionkey@0"
I20260619 16:49:37.871069 8139 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 16:49:37.872220 8154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:37.872499 8139 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:37.872606 8139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-1/wal
uuid: "0c052d73c57e481eaa3699c11d96116d"
format_stamp: "Formatted at 2026-06-19 16:49:37 on dist-test-slave-jsw9"
server_key: "86c51f324c28cf37adf281529e3c483e"
server_key_iv: "c615aafdf977ab70ee766f3fa7674746"
server_key_version: "encryptionkey@0"
I20260619 16:49:37.872720 8139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:37.893810 8139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:37.896605 8139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:37.896765 8139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:37.897715 8139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:37.897802 8139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:37.897826 8139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:37.897847 8139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:49:37 dist-test-slave-jsw9 krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887777, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 16:49:37.907234 8139 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38897
I20260619 16:49:37.907431 8269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38897 every 8 connection(s)
I20260619 16:49:37.907773 8139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:49:37.910295 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:37.898367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57605 (local address 127.0.98.62:37881)
0619 16:49:37.898463 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:37.898467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:37.898890 (+ 423us) server_negotiation.cc:407] Connection header received
0619 16:49:37.899779 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:37.899780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:37.899793 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:37.899847 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:37.901141 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.901819 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.903145 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.903484 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.903582 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:37.907046 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:37.907049 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:37.907049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:37.907062 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:37.908700 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:37.909210 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:37.909212 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:37.909213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:37.909277 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:37.909524 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:37.909525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:37.909525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:37.909672 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:37.909734 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:37.910055 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 16:49:37.910164 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:49:37.910490 8163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:37.898374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37881 (local address 127.0.98.2:57605)
0619 16:49:37.898668 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:37.898754 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:37.899630 (+ 876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:37.899962 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:37.899971 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:37.900439 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:37.900868 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:37.900876 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.901933 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.901935 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:37.903006 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:37.903011 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.903607 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.903608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:37.903801 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:37.904556 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:37.904567 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:37.906871 (+ 2304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:37.908807 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:37.908814 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:37.908817 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:37.909133 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:37.909338 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:37.909339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:37.909340 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:37.909458 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:37.909731 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:37.909737 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:37.909901 (+ 164us) client_negotiation.cc:769] Sending connection context
0619 16:49:37.910000 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 16:49:37.910238 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":100,"threads_started":1}
I20260619 16:49:37.911255 8270 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37881
I20260619 16:49:37.911404 8270 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:37.911633 8270 heartbeater.cc:511] Master 127.0.98.62:37881 requested a full tablet report, sending...
I20260619 16:49:37.912298 7927 ts_manager.cc:194] Registered new tserver with Master: 0c052d73c57e481eaa3699c11d96116d (127.0.98.2:38897)
I20260619 16:49:37.912989 7927 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57605
I20260619 16:49:37.913342 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 8139
I20260619 16:49:37.913484 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:49:37.913688 392 external_mini_cluster.cc:1428] Setting key acef35186602e51d87d8ab78b4166214
I20260619 16:49:37.922889 8276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:37.913006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37881 (local address 127.0.98.2:34779)
0619 16:49:37.914240 (+ 1234us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:37.914255 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:37.914604 (+ 349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:37.914794 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:37.914795 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:37.915023 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:37.915090 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:37.915093 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.916084 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.916085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:37.917134 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:37.917140 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.917832 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.917834 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:37.917967 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:37.918965 (+ 998us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:37.918972 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:37.919373 (+ 401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:37.921413 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:37.921415 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:37.921416 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:37.921648 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:37.921960 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:37.921962 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:37.921962 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:37.922037 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:37.922493 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:37.922494 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:37.922548 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 16:49:37.922623 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 16:49:37.922759 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1162,"thread_start_us":926,"threads_started":1}
I20260619 16:49:37.922889 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:37.913123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34779 (local address 127.0.98.62:37881)
0619 16:49:37.913196 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:37.913198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:37.914357 (+ 1159us) server_negotiation.cc:407] Connection header received
0619 16:49:37.914704 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:37.914705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:37.914718 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:37.914777 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:37.915168 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.915870 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.917485 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:37.917714 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:37.917795 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:37.919490 (+ 1695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:37.919493 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:37.919493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:37.919503 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:37.921284 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:37.921729 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:37.921731 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:37.921731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:37.921805 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:37.922107 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:37.922108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:37.922108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:37.922405 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:37.922485 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:37.922674 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 16:49:37.922786 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 16:49:37.951270 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37881
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:38.058133 8279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:38.058598 8279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:38.058691 8279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:38.060680 8279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:38.060771 8279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:38.060801 8279 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:49:38.060822 8279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:49:38.063205 8279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:38.063668 8279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:38.064021 8279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:38.070879 8287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:38.070885 8284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:38.070879 8285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:38.070950 8279 server_base.cc:1034] running on GCE node
I20260619 16:49:38.071384 8279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:38.071666 8279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:38.072829 8279 hybrid_clock.cc:648] HybridClock initialized: now 1781887778072817 us; error 32 us; skew 500 ppm
Jun 19 16:49:38 dist-test-slave-jsw9 krb5kdc[7882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887778, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:38.075829 8279 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 16:49:38.076467 8279 webserver.cc:460] Webserver started at http://127.0.98.3:39949/ using document root <none> and password file <none>
I20260619 16:49:38.076714 8279 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:38.076767 8279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:38.076916 8279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:38.078264 8279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-2/data/instance:
uuid: "1f1a774a25df46e18fe29d62a00b8bb9"
format_stamp: "Formatted at 2026-06-19 16:49:38 on dist-test-slave-jsw9"
server_key: "31a04392f8c38c88ad1af4af4913e797"
server_key_iv: "331834d625ba5cd8548653758d7c97c0"
server_key_version: "encryptionkey@0"
I20260619 16:49:38.078672 8279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1f1a774a25df46e18fe29d62a00b8bb9"
format_stamp: "Formatted at 2026-06-19 16:49:38 on dist-test-slave-jsw9"
server_key: "31a04392f8c38c88ad1af4af4913e797"
server_key_iv: "331834d625ba5cd8548653758d7c97c0"
server_key_version: "encryptionkey@0"
I20260619 16:49:38.080439 8279 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260619 16:49:38.081352 8294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:38.081583 8279 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:49:38.081702 8279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-2/wal
uuid: "1f1a774a25df46e18fe29d62a00b8bb9"
format_stamp: "Formatted at 2026-06-19 16:49:38 on dist-test-slave-jsw9"
server_key: "31a04392f8c38c88ad1af4af4913e797"
server_key_iv: "331834d625ba5cd8548653758d7c97c0"
server_key_version: "encryptionkey@0"
I20260619 16:49:38.081785 8279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:38.100431 8279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:38.102932 8279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:38.103099 8279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:38.103854 8279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:38.103911 8279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:38.103945 8279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:38.103960 8279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:49:38 dist-test-slave-jsw9 krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887778, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 16:49:38.112861 8279 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40539
I20260619 16:49:38.113035 8409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40539 every 8 connection(s)
I20260619 16:49:38.113361 8279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:49:38.116381 8303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:38.104489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37881 (local address 127.0.98.3:33383)
0619 16:49:38.104775 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:38.104837 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:38.105694 (+ 857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:38.106088 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:38.106096 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:38.106595 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:38.106992 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:38.107000 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.107806 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.107808 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:38.108977 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:38.108981 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.109388 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.109390 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:38.109526 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:38.110232 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:38.110255 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:38.112329 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:38.114518 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:38.114524 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:38.114526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:38.114804 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:38.115059 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:38.115059 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:38.115060 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:38.115221 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:38.115576 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:38.115582 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:38.115748 (+ 166us) client_negotiation.cc:769] Sending connection context
0619 16:49:38.115869 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 16:49:38.116046 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":96,"threads_started":1}
I20260619 16:49:38.116544 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:38.104463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33383 (local address 127.0.98.62:37881)
0619 16:49:38.104540 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:38.104543 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:38.104998 (+ 455us) server_negotiation.cc:407] Connection header received
0619 16:49:38.105863 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:38.105864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:38.105882 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:38.105960 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:38.107108 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.107695 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.109066 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.109292 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.109353 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:38.112456 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:38.112458 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:38.112459 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:38.112471 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:38.114405 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:38.114911 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:38.114912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:38.114912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:38.114982 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:38.115312 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:38.115319 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:38.115320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:38.115476 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:38.115554 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:38.116236 (+ 682us) server_negotiation.cc:299] Negotiation successful
0619 16:49:38.116473 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 16:49:38.117280 8410 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37881
I20260619 16:49:38.117383 8410 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:38.117599 8410 heartbeater.cc:511] Master 127.0.98.62:37881 requested a full tablet report, sending...
I20260619 16:49:38.118072 7927 ts_manager.cc:194] Registered new tserver with Master: 1f1a774a25df46e18fe29d62a00b8bb9 (127.0.98.3:40539)
I20260619 16:49:38.118515 7927 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33383
I20260619 16:49:38.118675 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 8279
I20260619 16:49:38.118793 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:49:38.119025 392 external_mini_cluster.cc:1428] Setting key 1b8a69b8d2e9a6a28730de856339cdbd
I20260619 16:49:38.119916 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:49:38.126401 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:38.118835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37881 (local address 127.0.98.3:54737)
0619 16:49:38.118984 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:38.118997 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:38.119054 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:38.119377 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:38.119378 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:38.119629 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:38.119704 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:38.119707 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.120859 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.120862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:38.121706 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:38.121711 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.122212 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.122213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:38.122292 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:38.122863 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:38.122868 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:38.123341 (+ 473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:38.125136 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:38.125138 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:38.125138 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:38.125379 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:38.125616 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:38.125617 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:38.125617 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:38.125689 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:38.125998 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:38.126000 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:38.126056 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 16:49:38.126130 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 16:49:38.126298 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":90,"threads_started":1}
I20260619 16:49:38.126413 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:38.119212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54737 (local address 127.0.98.62:37881)
0619 16:49:38.119269 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:38.119271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:38.119279 (+ 8us) server_negotiation.cc:407] Connection header received
0619 16:49:38.119301 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:38.119302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:38.119312 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:38.119357 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:38.119774 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.120503 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.121813 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.122100 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.122173 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:38.123456 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:38.123458 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:38.123458 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:38.123467 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:38.125031 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:38.125459 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:38.125460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:38.125460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:38.125529 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:38.125754 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:38.125755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:38.125755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:38.125911 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:38.125985 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:38.126172 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 16:49:38.126313 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
Jun 19 16:49:38 dist-test-slave-jsw9 krb5kdc[7882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887778, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 16:49:38 dist-test-slave-jsw9 krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:38 dist-test-slave-jsw9 krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 16:49:38.145753 7976 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:38.139488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54452 (local address 127.0.98.62:37881)
0619 16:49:38.139549 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:38.139551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:38.139558 (+ 7us) server_negotiation.cc:407] Connection header received
0619 16:49:38.139586 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:38.139587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:38.139596 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:38.139647 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:38.140306 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.140857 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.141632 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.141832 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.141894 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:38.145650 (+ 3756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54452: BlockingRecv error: recv got EOF from 127.0.0.1:54452 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 19 16:49:38 dist-test-slave-jsw9 krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887778, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 16:49:38.154217 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:38.146640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54456 (local address 127.0.98.62:37881)
0619 16:49:38.146694 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:38.146697 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:38.146793 (+ 96us) server_negotiation.cc:407] Connection header received
0619 16:49:38.146851 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:38.146852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:38.146862 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:38.146928 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:38.147494 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.148070 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.148793 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.148987 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.149046 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:38.151383 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:38.151385 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:38.151385 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:38.151395 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:38.152931 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:38.153396 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:38.153397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:38.153397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:38.153464 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:38.153672 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:38.153672 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:38.153673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:38.153799 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:38.153894 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:38.153979 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 16:49:38.154089 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 16:49:38.156212 7927 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:49:38.158639 7927 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:49:38.167402 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:38.161605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40539 (local address 127.0.0.1:45036)
0619 16:49:38.162232 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:38.162245 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:38.162307 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:38.162670 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:38.162671 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:38.162673 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:38.162711 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:38.162712 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.165232 (+ 2520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.165234 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:38.166008 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:38.166010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.166989 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.166991 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:38.167100 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:38.167118 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 16:49:38.167191 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 16:49:38.167262 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":5,"thread_start_us":101,"threads_started":1}
I20260619 16:49:38.167456 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:38.161852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60088 (local address 127.0.98.1:37025)
0619 16:49:38.162060 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:38.162067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:38.162081 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:49:38.162274 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:38.162278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:38.162297 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:38.162375 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:38.162790 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.164122 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.166011 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.166889 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.166968 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:38.167068 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:38.167299 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 16:49:38.167350 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":87,"threads_started":1}
I20260619 16:49:38.167634 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:38.162024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48358 (local address 127.0.98.2:38897)
0619 16:49:38.162259 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:38.162263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:38.162512 (+ 249us) server_negotiation.cc:407] Connection header received
0619 16:49:38.162557 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:38.162559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:38.162575 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:38.162645 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:38.162798 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.164123 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.166264 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.167096 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.167160 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:38.167239 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:38.167459 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 16:49:38.167515 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":105,"threads_started":1}
I20260619 16:49:38.168089 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:38.161743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45036 (local address 127.0.98.3:40539)
0619 16:49:38.162302 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:38.162307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:38.162318 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:49:38.162383 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:38.162385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:38.162403 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:38.162496 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:38.164258 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.165139 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.166098 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.166761 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.166851 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:38.166956 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:38.167942 (+ 986us) server_negotiation.cc:299] Negotiation successful
0619 16:49:38.167995 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":108,"threads_started":1}
I20260619 16:49:38.167414 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:38.161606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37025 (local address 127.0.0.1:60088)
0619 16:49:38.161974 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:38.162022 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:38.162099 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:38.162516 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:38.162520 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:38.162523 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:38.162705 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:38.162708 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.164220 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.164222 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:38.165890 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:38.165892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.167108 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.167109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:38.167203 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:38.167207 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:49:38.167265 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 16:49:38.167324 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":98,"thread_start_us":100,"threads_started":1}
I20260619 16:49:38.167609 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:38.161876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38897 (local address 127.0.0.1:48358)
0619 16:49:38.162396 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:38.162404 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:38.162457 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:38.162661 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:38.162662 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:38.162664 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:38.162715 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:38.162718 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.164223 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.164224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:38.165887 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:38.165892 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.167273 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.167274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:38.167328 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:38.167332 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:49:38.167375 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 16:49:38.167418 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":48,"thread_start_us":73,"threads_started":1}
I20260619 16:49:38.169077 8064 tablet_service.cc:1467] Processing CreateTablet for tablet e1d3931fa7184ae886e0afb7abadbc9e (DEFAULT_TABLE table=test-table [id=eda40069f55e4e9b8854c2caa59450bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:38.169083 8344 tablet_service.cc:1467] Processing CreateTablet for tablet e1d3931fa7184ae886e0afb7abadbc9e (DEFAULT_TABLE table=test-table [id=eda40069f55e4e9b8854c2caa59450bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:38.169216 8204 tablet_service.cc:1467] Processing CreateTablet for tablet e1d3931fa7184ae886e0afb7abadbc9e (DEFAULT_TABLE table=test-table [id=eda40069f55e4e9b8854c2caa59450bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:38.169466 8064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1d3931fa7184ae886e0afb7abadbc9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:38.169469 8344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1d3931fa7184ae886e0afb7abadbc9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:38.169516 8204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1d3931fa7184ae886e0afb7abadbc9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:38.173060 8438 tablet_bootstrap.cc:492] T e1d3931fa7184ae886e0afb7abadbc9e P edba86e2b7a74ab880b1f716db2c7471: Bootstrap starting.
I20260619 16:49:38.174048 8439 tablet_bootstrap.cc:492] T e1d3931fa7184ae886e0afb7abadbc9e P 1f1a774a25df46e18fe29d62a00b8bb9: Bootstrap starting.
I20260619 16:49:38.174444 8440 tablet_bootstrap.cc:492] T e1d3931fa7184ae886e0afb7abadbc9e P 0c052d73c57e481eaa3699c11d96116d: Bootstrap starting.
I20260619 16:49:38.174651 8438 tablet_bootstrap.cc:654] T e1d3931fa7184ae886e0afb7abadbc9e P edba86e2b7a74ab880b1f716db2c7471: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:38.175045 8439 tablet_bootstrap.cc:654] T e1d3931fa7184ae886e0afb7abadbc9e P 1f1a774a25df46e18fe29d62a00b8bb9: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:38.175060 8440 tablet_bootstrap.cc:654] T e1d3931fa7184ae886e0afb7abadbc9e P 0c052d73c57e481eaa3699c11d96116d: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:38.175308 8438 log.cc:824] T e1d3931fa7184ae886e0afb7abadbc9e P edba86e2b7a74ab880b1f716db2c7471: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:38.175402 8440 log.cc:824] T e1d3931fa7184ae886e0afb7abadbc9e P 0c052d73c57e481eaa3699c11d96116d: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:38.175442 8439 log.cc:824] T e1d3931fa7184ae886e0afb7abadbc9e P 1f1a774a25df46e18fe29d62a00b8bb9: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:38.176247 8440 tablet_bootstrap.cc:492] T e1d3931fa7184ae886e0afb7abadbc9e P 0c052d73c57e481eaa3699c11d96116d: No bootstrap required, opened a new log
I20260619 16:49:38.176347 8440 ts_tablet_manager.cc:1397] T e1d3931fa7184ae886e0afb7abadbc9e P 0c052d73c57e481eaa3699c11d96116d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:49:38.176460 8439 tablet_bootstrap.cc:492] T e1d3931fa7184ae886e0afb7abadbc9e P 1f1a774a25df46e18fe29d62a00b8bb9: No bootstrap required, opened a new log
I20260619 16:49:38.176564 8439 ts_tablet_manager.cc:1397] T e1d3931fa7184ae886e0afb7abadbc9e P 1f1a774a25df46e18fe29d62a00b8bb9: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 16:49:38.176525 8438 tablet_bootstrap.cc:492] T e1d3931fa7184ae886e0afb7abadbc9e P edba86e2b7a74ab880b1f716db2c7471: No bootstrap required, opened a new log
I20260619 16:49:38.176633 8438 ts_tablet_manager.cc:1397] T e1d3931fa7184ae886e0afb7abadbc9e P edba86e2b7a74ab880b1f716db2c7471: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 16:49:38.178313 8439 raft_consensus.cc:348] T e1d3931fa7184ae886e0afb7abadbc9e P 1f1a774a25df46e18fe29d62a00b8bb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } } peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } }
I20260619 16:49:38.178544 8439 raft_consensus.cc:374] T e1d3931fa7184ae886e0afb7abadbc9e P 1f1a774a25df46e18fe29d62a00b8bb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:38.178581 8439 raft_consensus.cc:729] T e1d3931fa7184ae886e0afb7abadbc9e P 1f1a774a25df46e18fe29d62a00b8bb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f1a774a25df46e18fe29d62a00b8bb9, State: Initialized, Role: FOLLOWER
I20260619 16:49:38.178737 8439 consensus_queue.cc:260] T e1d3931fa7184ae886e0afb7abadbc9e P 1f1a774a25df46e18fe29d62a00b8bb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } } peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } }
I20260619 16:49:38.178793 8440 raft_consensus.cc:348] T e1d3931fa7184ae886e0afb7abadbc9e P 0c052d73c57e481eaa3699c11d96116d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } } peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } }
I20260619 16:49:38.178975 8440 raft_consensus.cc:374] T e1d3931fa7184ae886e0afb7abadbc9e P 0c052d73c57e481eaa3699c11d96116d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:38.179014 8440 raft_consensus.cc:729] T e1d3931fa7184ae886e0afb7abadbc9e P 0c052d73c57e481eaa3699c11d96116d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c052d73c57e481eaa3699c11d96116d, State: Initialized, Role: FOLLOWER
I20260619 16:49:38.178998 8438 raft_consensus.cc:348] T e1d3931fa7184ae886e0afb7abadbc9e P edba86e2b7a74ab880b1f716db2c7471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } } peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } }
I20260619 16:49:38.179139 8438 raft_consensus.cc:374] T e1d3931fa7184ae886e0afb7abadbc9e P edba86e2b7a74ab880b1f716db2c7471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:38.179171 8438 raft_consensus.cc:729] T e1d3931fa7184ae886e0afb7abadbc9e P edba86e2b7a74ab880b1f716db2c7471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edba86e2b7a74ab880b1f716db2c7471, State: Initialized, Role: FOLLOWER
I20260619 16:49:38.179177 8439 ts_tablet_manager.cc:1428] T e1d3931fa7184ae886e0afb7abadbc9e P 1f1a774a25df46e18fe29d62a00b8bb9: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260619 16:49:38.179168 8440 consensus_queue.cc:260] T e1d3931fa7184ae886e0afb7abadbc9e P 0c052d73c57e481eaa3699c11d96116d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } } peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } }
I20260619 16:49:38.179306 8438 consensus_queue.cc:260] T e1d3931fa7184ae886e0afb7abadbc9e P edba86e2b7a74ab880b1f716db2c7471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } } peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } }
I20260619 16:49:38.179371 8410 heartbeater.cc:503] Master 127.0.98.62:37881 was elected leader, sending a full tablet report...
I20260619 16:49:38.179464 8440 ts_tablet_manager.cc:1428] T e1d3931fa7184ae886e0afb7abadbc9e P 0c052d73c57e481eaa3699c11d96116d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 16:49:38.179565 8438 ts_tablet_manager.cc:1428] T e1d3931fa7184ae886e0afb7abadbc9e P edba86e2b7a74ab880b1f716db2c7471: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 16:49:38.179579 8270 heartbeater.cc:503] Master 127.0.98.62:37881 was elected leader, sending a full tablet report...
I20260619 16:49:38.179610 8130 heartbeater.cc:503] Master 127.0.98.62:37881 was elected leader, sending a full tablet report...
W20260619 16:49:38.202412 8131 tablet.cc:2369] T e1d3931fa7184ae886e0afb7abadbc9e P edba86e2b7a74ab880b1f716db2c7471: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:49:38.363951 8445 raft_consensus.cc:482] T e1d3931fa7184ae886e0afb7abadbc9e P 0c052d73c57e481eaa3699c11d96116d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:38.364173 8445 raft_consensus.cc:504] T e1d3931fa7184ae886e0afb7abadbc9e P 0c052d73c57e481eaa3699c11d96116d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } } peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } }
W20260619 16:49:38.364249 8411 tablet.cc:2369] T e1d3931fa7184ae886e0afb7abadbc9e P 1f1a774a25df46e18fe29d62a00b8bb9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:49:38.364670 8445 leader_election.cc:288] T e1d3931fa7184ae886e0afb7abadbc9e P 0c052d73c57e481eaa3699c11d96116d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f1a774a25df46e18fe29d62a00b8bb9 (127.0.98.3:40539), edba86e2b7a74ab880b1f716db2c7471 (127.0.98.1:37025)
I20260619 16:49:38.369421 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:38.365014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57127 (local address 127.0.98.1:37025)
0619 16:49:38.365066 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:38.365069 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:38.365270 (+ 201us) server_negotiation.cc:407] Connection header received
0619 16:49:38.365333 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:38.365334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:38.365349 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:38.365401 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:38.365547 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.366365 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.368070 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.368718 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.368783 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:38.368793 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:38.369262 (+ 469us) server_negotiation.cc:299] Negotiation successful
0619 16:49:38.369325 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 16:49:38.369421 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:38.365050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60309 (local address 127.0.98.3:40539)
0619 16:49:38.365099 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:38.365101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:38.365115 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:49:38.365245 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:38.365246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:38.365262 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:38.365314 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:38.365521 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.366364 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.368070 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.368685 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.368775 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:38.368792 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:38.369262 (+ 470us) server_negotiation.cc:299] Negotiation successful
0619 16:49:38.369324 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 16:49:38.369496 8163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:38.364939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40539 (local address 127.0.98.2:60309)
0619 16:49:38.365013 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:38.365028 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:38.365091 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:38.365348 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:38.365350 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:38.365352 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:38.365439 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:38.365441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.366551 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.366552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:38.367941 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:38.367943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.368894 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.368897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:38.369077 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:38.369089 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 16:49:38.369220 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 16:49:38.369288 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":94}
I20260619 16:49:38.369518 8447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:38.364939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37025 (local address 127.0.98.2:57127)
0619 16:49:38.365194 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:38.365203 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:38.365277 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:38.365414 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:38.365414 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:38.365416 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:38.365483 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:38.365484 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.366551 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.366552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:38.367937 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:38.367943 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.368929 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.368930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:38.369077 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:38.369090 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 16:49:38.369220 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 16:49:38.369288 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":78,"thread_start_us":129,"threads_started":1}
I20260619 16:49:38.369957 8364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1d3931fa7184ae886e0afb7abadbc9e" candidate_uuid: "0c052d73c57e481eaa3699c11d96116d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" is_pre_election: true
I20260619 16:49:38.369947 8084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1d3931fa7184ae886e0afb7abadbc9e" candidate_uuid: "0c052d73c57e481eaa3699c11d96116d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edba86e2b7a74ab880b1f716db2c7471" is_pre_election: true
I20260619 16:49:38.370170 8084 raft_consensus.cc:2454] T e1d3931fa7184ae886e0afb7abadbc9e P edba86e2b7a74ab880b1f716db2c7471 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c052d73c57e481eaa3699c11d96116d in term 0.
I20260619 16:49:38.370170 8364 raft_consensus.cc:2454] T e1d3931fa7184ae886e0afb7abadbc9e P 1f1a774a25df46e18fe29d62a00b8bb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c052d73c57e481eaa3699c11d96116d in term 0.
I20260619 16:49:38.370517 8158 leader_election.cc:302] T e1d3931fa7184ae886e0afb7abadbc9e P 0c052d73c57e481eaa3699c11d96116d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c052d73c57e481eaa3699c11d96116d, 1f1a774a25df46e18fe29d62a00b8bb9; no voters:
I20260619 16:49:38.370735 8445 raft_consensus.cc:2781] T e1d3931fa7184ae886e0afb7abadbc9e P 0c052d73c57e481eaa3699c11d96116d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:49:38.370814 8445 raft_consensus.cc:482] T e1d3931fa7184ae886e0afb7abadbc9e P 0c052d73c57e481eaa3699c11d96116d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:49:38.370841 8445 raft_consensus.cc:3037] T e1d3931fa7184ae886e0afb7abadbc9e P 0c052d73c57e481eaa3699c11d96116d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:38.372054 8445 raft_consensus.cc:504] T e1d3931fa7184ae886e0afb7abadbc9e P 0c052d73c57e481eaa3699c11d96116d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } } peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } }
I20260619 16:49:38.372246 8445 leader_election.cc:288] T e1d3931fa7184ae886e0afb7abadbc9e P 0c052d73c57e481eaa3699c11d96116d [CANDIDATE]: Term 1 election: Requested vote from peers 1f1a774a25df46e18fe29d62a00b8bb9 (127.0.98.3:40539), edba86e2b7a74ab880b1f716db2c7471 (127.0.98.1:37025)
I20260619 16:49:38.372462 8084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1d3931fa7184ae886e0afb7abadbc9e" candidate_uuid: "0c052d73c57e481eaa3699c11d96116d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edba86e2b7a74ab880b1f716db2c7471"
I20260619 16:49:38.372470 8364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1d3931fa7184ae886e0afb7abadbc9e" candidate_uuid: "0c052d73c57e481eaa3699c11d96116d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f1a774a25df46e18fe29d62a00b8bb9"
I20260619 16:49:38.372566 8364 raft_consensus.cc:3037] T e1d3931fa7184ae886e0afb7abadbc9e P 1f1a774a25df46e18fe29d62a00b8bb9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:38.372570 8084 raft_consensus.cc:3037] T e1d3931fa7184ae886e0afb7abadbc9e P edba86e2b7a74ab880b1f716db2c7471 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:38.373469 8084 raft_consensus.cc:2454] T e1d3931fa7184ae886e0afb7abadbc9e P edba86e2b7a74ab880b1f716db2c7471 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c052d73c57e481eaa3699c11d96116d in term 1.
I20260619 16:49:38.373497 8364 raft_consensus.cc:2454] T e1d3931fa7184ae886e0afb7abadbc9e P 1f1a774a25df46e18fe29d62a00b8bb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c052d73c57e481eaa3699c11d96116d in term 1.
I20260619 16:49:38.373667 8158 leader_election.cc:302] T e1d3931fa7184ae886e0afb7abadbc9e P 0c052d73c57e481eaa3699c11d96116d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c052d73c57e481eaa3699c11d96116d, 1f1a774a25df46e18fe29d62a00b8bb9; no voters:
I20260619 16:49:38.373819 8445 raft_consensus.cc:2781] T e1d3931fa7184ae886e0afb7abadbc9e P 0c052d73c57e481eaa3699c11d96116d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:38.373943 8445 raft_consensus.cc:686] T e1d3931fa7184ae886e0afb7abadbc9e P 0c052d73c57e481eaa3699c11d96116d [term 1 LEADER]: Becoming Leader. State: Replica: 0c052d73c57e481eaa3699c11d96116d, State: Running, Role: LEADER
I20260619 16:49:38.374063 8445 consensus_queue.cc:237] T e1d3931fa7184ae886e0afb7abadbc9e P 0c052d73c57e481eaa3699c11d96116d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } } peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } }
I20260619 16:49:38.375226 7925 catalog_manager.cc:5495] T e1d3931fa7184ae886e0afb7abadbc9e P 0c052d73c57e481eaa3699c11d96116d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c052d73c57e481eaa3699c11d96116d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0c052d73c57e481eaa3699c11d96116d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } health_report { overall_health: HEALTHY } } }
I20260619 16:49:38.397801 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:38.395180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48362 (local address 127.0.98.2:38897)
0619 16:49:38.395243 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:38.395245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:38.395257 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:49:38.395299 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:38.395300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:38.395323 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:38.395397 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:38.395526 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.396037 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.396906 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.397152 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.397220 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:38.397428 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:38.397567 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:38.397665 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:38.397678 (+ 13us) server_negotiation.cc:299] Negotiation successful
0619 16:49:38.397704 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 16:49:38.400489 8364 raft_consensus.cc:1261] T e1d3931fa7184ae886e0afb7abadbc9e P 1f1a774a25df46e18fe29d62a00b8bb9 [term 1 FOLLOWER]: Refusing update from remote peer 0c052d73c57e481eaa3699c11d96116d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:38.400489 8084 raft_consensus.cc:1261] T e1d3931fa7184ae886e0afb7abadbc9e P edba86e2b7a74ab880b1f716db2c7471 [term 1 FOLLOWER]: Refusing update from remote peer 0c052d73c57e481eaa3699c11d96116d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:38.401073 8448 consensus_queue.cc:1035] T e1d3931fa7184ae886e0afb7abadbc9e P 0c052d73c57e481eaa3699c11d96116d [LEADER]: Connected to new peer: Peer: permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:38.401285 8445 consensus_queue.cc:1035] T e1d3931fa7184ae886e0afb7abadbc9e P 0c052d73c57e481eaa3699c11d96116d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:38.405403 8453 mvcc.cc:205] Tried to move back new op lower bound from 7298612340324831232 to 7298612340220977152. Current Snapshot: MvccSnapshot[applied={T|T < 7298612340324831232}]
I20260619 16:49:38.407143 7926 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:38.407265 7926 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:38.408385 7926 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 384f080274ce451a8180fbc2fbda74ec: Sending DeleteTablet for 3 replicas of tablet e1d3931fa7184ae886e0afb7abadbc9e
I20260619 16:49:38.408839 8204 tablet_service.cc:1514] Processing DeleteTablet for tablet e1d3931fa7184ae886e0afb7abadbc9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:38 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48358
I20260619 16:49:38.408859 8064 tablet_service.cc:1514] Processing DeleteTablet for tablet e1d3931fa7184ae886e0afb7abadbc9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:38 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60088
I20260619 16:49:38.409307 7926 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 16:49:38.409492 8460 tablet_replica.cc:330] T e1d3931fa7184ae886e0afb7abadbc9e P edba86e2b7a74ab880b1f716db2c7471: stopping tablet replica
I20260619 16:49:38.409482 8344 tablet_service.cc:1514] Processing DeleteTablet for tablet e1d3931fa7184ae886e0afb7abadbc9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:38 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45036
W20260619 16:49:38.409540 7926 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:49:38.409633 8460 raft_consensus.cc:2229] T e1d3931fa7184ae886e0afb7abadbc9e P edba86e2b7a74ab880b1f716db2c7471 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:38.409778 8460 raft_consensus.cc:2258] T e1d3931fa7184ae886e0afb7abadbc9e P edba86e2b7a74ab880b1f716db2c7471 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:38.410384 8460 ts_tablet_manager.cc:1905] T e1d3931fa7184ae886e0afb7abadbc9e P edba86e2b7a74ab880b1f716db2c7471: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:38.411150 8344 tablet_service.cc:1467] Processing CreateTablet for tablet 160e94ea1df443f8bd7a00ac220953c0 (DEFAULT_TABLE table=test-table [id=2cbcd2a91e094e4ebf2b992b3e6da07d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:38.411252 8462 tablet_replica.cc:330] T e1d3931fa7184ae886e0afb7abadbc9e P 1f1a774a25df46e18fe29d62a00b8bb9: stopping tablet replica
I20260619 16:49:38.411248 8064 tablet_service.cc:1467] Processing CreateTablet for tablet 160e94ea1df443f8bd7a00ac220953c0 (DEFAULT_TABLE table=test-table [id=2cbcd2a91e094e4ebf2b992b3e6da07d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:38.411294 8344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 160e94ea1df443f8bd7a00ac220953c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:38.411360 8462 raft_consensus.cc:2229] T e1d3931fa7184ae886e0afb7abadbc9e P 1f1a774a25df46e18fe29d62a00b8bb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:38.411372 8064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 160e94ea1df443f8bd7a00ac220953c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:38.411494 8462 raft_consensus.cc:2258] T e1d3931fa7184ae886e0afb7abadbc9e P 1f1a774a25df46e18fe29d62a00b8bb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:38.412791 8439 tablet_bootstrap.cc:492] T 160e94ea1df443f8bd7a00ac220953c0 P 1f1a774a25df46e18fe29d62a00b8bb9: Bootstrap starting.
I20260619 16:49:38.412989 8438 tablet_bootstrap.cc:492] T 160e94ea1df443f8bd7a00ac220953c0 P edba86e2b7a74ab880b1f716db2c7471: Bootstrap starting.
I20260619 16:49:38.413322 8439 tablet_bootstrap.cc:654] T 160e94ea1df443f8bd7a00ac220953c0 P 1f1a774a25df46e18fe29d62a00b8bb9: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:38.413434 8438 tablet_bootstrap.cc:654] T 160e94ea1df443f8bd7a00ac220953c0 P edba86e2b7a74ab880b1f716db2c7471: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:38.413434 8204 tablet_service.cc:1467] Processing CreateTablet for tablet 160e94ea1df443f8bd7a00ac220953c0 (DEFAULT_TABLE table=test-table [id=2cbcd2a91e094e4ebf2b992b3e6da07d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:38.413582 8204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 160e94ea1df443f8bd7a00ac220953c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:38.414268 8461 tablet_replica.cc:330] T e1d3931fa7184ae886e0afb7abadbc9e P 0c052d73c57e481eaa3699c11d96116d: stopping tablet replica
I20260619 16:49:38.414003 8462 ts_tablet_manager.cc:1905] T e1d3931fa7184ae886e0afb7abadbc9e P 1f1a774a25df46e18fe29d62a00b8bb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:38.414479 8461 raft_consensus.cc:2229] T e1d3931fa7184ae886e0afb7abadbc9e P 0c052d73c57e481eaa3699c11d96116d [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:49:38.414660 8461 raft_consensus.cc:2258] T e1d3931fa7184ae886e0afb7abadbc9e P 0c052d73c57e481eaa3699c11d96116d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:38.414866 8460 ts_tablet_manager.cc:1918] T e1d3931fa7184ae886e0afb7abadbc9e P edba86e2b7a74ab880b1f716db2c7471: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:38.414976 8460 log.cc:1196] T e1d3931fa7184ae886e0afb7abadbc9e P edba86e2b7a74ab880b1f716db2c7471: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-0/wal/wals/e1d3931fa7184ae886e0afb7abadbc9e
I20260619 16:49:38.415298 8460 ts_tablet_manager.cc:1939] T e1d3931fa7184ae886e0afb7abadbc9e P edba86e2b7a74ab880b1f716db2c7471: Deleting consensus metadata
I20260619 16:49:38.415534 8440 tablet_bootstrap.cc:492] T 160e94ea1df443f8bd7a00ac220953c0 P 0c052d73c57e481eaa3699c11d96116d: Bootstrap starting.
I20260619 16:49:38.415794 7913 catalog_manager.cc:4841] TS edba86e2b7a74ab880b1f716db2c7471 (127.0.98.1:37025): tablet e1d3931fa7184ae886e0afb7abadbc9e (table test-table [id=eda40069f55e4e9b8854c2caa59450bb]) successfully deleted
I20260619 16:49:38.415997 8440 tablet_bootstrap.cc:654] T 160e94ea1df443f8bd7a00ac220953c0 P 0c052d73c57e481eaa3699c11d96116d: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:38.416152 8462 ts_tablet_manager.cc:1918] T e1d3931fa7184ae886e0afb7abadbc9e P 1f1a774a25df46e18fe29d62a00b8bb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:38.416216 8462 log.cc:1196] T e1d3931fa7184ae886e0afb7abadbc9e P 1f1a774a25df46e18fe29d62a00b8bb9: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-2/wal/wals/e1d3931fa7184ae886e0afb7abadbc9e
I20260619 16:49:38.416334 8461 ts_tablet_manager.cc:1905] T e1d3931fa7184ae886e0afb7abadbc9e P 0c052d73c57e481eaa3699c11d96116d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:38.416822 8462 ts_tablet_manager.cc:1939] T e1d3931fa7184ae886e0afb7abadbc9e P 1f1a774a25df46e18fe29d62a00b8bb9: Deleting consensus metadata
I20260619 16:49:38.417268 8440 tablet_bootstrap.cc:492] T 160e94ea1df443f8bd7a00ac220953c0 P 0c052d73c57e481eaa3699c11d96116d: No bootstrap required, opened a new log
I20260619 16:49:38.417316 8440 ts_tablet_manager.cc:1397] T 160e94ea1df443f8bd7a00ac220953c0 P 0c052d73c57e481eaa3699c11d96116d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:49:38.417310 7914 catalog_manager.cc:4841] TS 1f1a774a25df46e18fe29d62a00b8bb9 (127.0.98.3:40539): tablet e1d3931fa7184ae886e0afb7abadbc9e (table test-table [id=eda40069f55e4e9b8854c2caa59450bb]) successfully deleted
I20260619 16:49:38.417515 8440 raft_consensus.cc:348] T 160e94ea1df443f8bd7a00ac220953c0 P 0c052d73c57e481eaa3699c11d96116d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } }
I20260619 16:49:38.417596 8440 raft_consensus.cc:374] T 160e94ea1df443f8bd7a00ac220953c0 P 0c052d73c57e481eaa3699c11d96116d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:38.417615 8440 raft_consensus.cc:729] T 160e94ea1df443f8bd7a00ac220953c0 P 0c052d73c57e481eaa3699c11d96116d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c052d73c57e481eaa3699c11d96116d, State: Initialized, Role: FOLLOWER
I20260619 16:49:38.417662 8440 consensus_queue.cc:260] T 160e94ea1df443f8bd7a00ac220953c0 P 0c052d73c57e481eaa3699c11d96116d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } }
I20260619 16:49:38.417755 8440 ts_tablet_manager.cc:1428] T 160e94ea1df443f8bd7a00ac220953c0 P 0c052d73c57e481eaa3699c11d96116d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:38.417773 8438 tablet_bootstrap.cc:492] T 160e94ea1df443f8bd7a00ac220953c0 P edba86e2b7a74ab880b1f716db2c7471: No bootstrap required, opened a new log
I20260619 16:49:38.417814 8438 ts_tablet_manager.cc:1397] T 160e94ea1df443f8bd7a00ac220953c0 P edba86e2b7a74ab880b1f716db2c7471: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.000s
I20260619 16:49:38.417872 8461 ts_tablet_manager.cc:1918] T e1d3931fa7184ae886e0afb7abadbc9e P 0c052d73c57e481eaa3699c11d96116d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:38.417935 8461 log.cc:1196] T e1d3931fa7184ae886e0afb7abadbc9e P 0c052d73c57e481eaa3699c11d96116d: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-1/wal/wals/e1d3931fa7184ae886e0afb7abadbc9e
I20260619 16:49:38.417972 8438 raft_consensus.cc:348] T 160e94ea1df443f8bd7a00ac220953c0 P edba86e2b7a74ab880b1f716db2c7471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } }
I20260619 16:49:38.418047 8438 raft_consensus.cc:374] T 160e94ea1df443f8bd7a00ac220953c0 P edba86e2b7a74ab880b1f716db2c7471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:38.418063 8438 raft_consensus.cc:729] T 160e94ea1df443f8bd7a00ac220953c0 P edba86e2b7a74ab880b1f716db2c7471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edba86e2b7a74ab880b1f716db2c7471, State: Initialized, Role: FOLLOWER
I20260619 16:49:38.418108 8438 consensus_queue.cc:260] T 160e94ea1df443f8bd7a00ac220953c0 P edba86e2b7a74ab880b1f716db2c7471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } }
I20260619 16:49:38.418151 8461 ts_tablet_manager.cc:1939] T e1d3931fa7184ae886e0afb7abadbc9e P 0c052d73c57e481eaa3699c11d96116d: Deleting consensus metadata
I20260619 16:49:38.418356 8438 ts_tablet_manager.cc:1428] T 160e94ea1df443f8bd7a00ac220953c0 P edba86e2b7a74ab880b1f716db2c7471: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:38.418601 8439 tablet_bootstrap.cc:492] T 160e94ea1df443f8bd7a00ac220953c0 P 1f1a774a25df46e18fe29d62a00b8bb9: No bootstrap required, opened a new log
I20260619 16:49:38.418649 8439 ts_tablet_manager.cc:1397] T 160e94ea1df443f8bd7a00ac220953c0 P 1f1a774a25df46e18fe29d62a00b8bb9: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.000s
I20260619 16:49:38.418848 8439 raft_consensus.cc:348] T 160e94ea1df443f8bd7a00ac220953c0 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } }
I20260619 16:49:38.418927 8439 raft_consensus.cc:374] T 160e94ea1df443f8bd7a00ac220953c0 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:38.418948 8439 raft_consensus.cc:729] T 160e94ea1df443f8bd7a00ac220953c0 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f1a774a25df46e18fe29d62a00b8bb9, State: Initialized, Role: FOLLOWER
I20260619 16:49:38.418991 8439 consensus_queue.cc:260] T 160e94ea1df443f8bd7a00ac220953c0 P 1f1a774a25df46e18fe29d62a00b8bb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } }
I20260619 16:49:38.419080 8439 ts_tablet_manager.cc:1428] T 160e94ea1df443f8bd7a00ac220953c0 P 1f1a774a25df46e18fe29d62a00b8bb9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:38.419595 7913 catalog_manager.cc:4841] TS 0c052d73c57e481eaa3699c11d96116d (127.0.98.2:38897): tablet e1d3931fa7184ae886e0afb7abadbc9e (table test-table [id=eda40069f55e4e9b8854c2caa59450bb]) successfully deleted
I20260619 16:49:38.632871 8444 raft_consensus.cc:482] T 160e94ea1df443f8bd7a00ac220953c0 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:38.633035 8444 raft_consensus.cc:504] T 160e94ea1df443f8bd7a00ac220953c0 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } }
I20260619 16:49:38.633424 8444 leader_election.cc:288] T 160e94ea1df443f8bd7a00ac220953c0 P 1f1a774a25df46e18fe29d62a00b8bb9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edba86e2b7a74ab880b1f716db2c7471 (127.0.98.1:37025), 0c052d73c57e481eaa3699c11d96116d (127.0.98.2:38897)
I20260619 16:49:38.636773 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:38.633777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58655 (local address 127.0.98.1:37025)
0619 16:49:38.633842 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:38.633844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:38.633920 (+ 76us) server_negotiation.cc:407] Connection header received
0619 16:49:38.634079 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:38.634081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:38.634091 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:38.634123 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:38.634375 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.634904 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.636012 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.636414 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.636480 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:38.636495 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:38.636637 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 16:49:38.636692 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 16:49:38.636806 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:38.633981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43461 (local address 127.0.98.2:38897)
0619 16:49:38.634047 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:38.634049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:38.634064 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:49:38.634161 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:38.634162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:38.634171 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:38.634284 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:38.634427 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.634918 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.636023 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.636460 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.636523 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:38.636536 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:38.636675 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 16:49:38.636728 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 16:49:38.636833 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:38.633886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38897 (local address 127.0.98.3:43461)
0619 16:49:38.633998 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:38.634009 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:38.634081 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:38.634288 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:38.634289 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:38.634292 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:38.634361 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:38.634362 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.634997 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.634998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:38.635918 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:38.635920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.636534 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.636535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:38.636609 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:38.636612 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:49:38.636659 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 16:49:38.636710 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":77,"thread_start_us":67,"threads_started":1}
I20260619 16:49:38.636834 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:38.633672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37025 (local address 127.0.98.3:58655)
0619 16:49:38.633824 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:38.633837 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:38.633935 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:38.634224 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:38.634225 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:38.634228 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:38.634308 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:38.634311 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.634997 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.634998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:38.635907 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:38.635910 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.636491 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.636492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:38.636569 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:38.636574 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:49:38.636623 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 16:49:38.636680 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":2,"thread_start_us":87,"threads_started":1}
I20260619 16:49:38.637138 8224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "160e94ea1df443f8bd7a00ac220953c0" candidate_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c052d73c57e481eaa3699c11d96116d" is_pre_election: true
I20260619 16:49:38.637337 8224 raft_consensus.cc:2454] T 160e94ea1df443f8bd7a00ac220953c0 P 0c052d73c57e481eaa3699c11d96116d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f1a774a25df46e18fe29d62a00b8bb9 in term 0.
I20260619 16:49:38.637574 8297 leader_election.cc:302] T 160e94ea1df443f8bd7a00ac220953c0 P 1f1a774a25df46e18fe29d62a00b8bb9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c052d73c57e481eaa3699c11d96116d, 1f1a774a25df46e18fe29d62a00b8bb9; no voters:
I20260619 16:49:38.637805 8444 raft_consensus.cc:2781] T 160e94ea1df443f8bd7a00ac220953c0 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:49:38.637877 8444 raft_consensus.cc:482] T 160e94ea1df443f8bd7a00ac220953c0 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:49:38.637909 8444 raft_consensus.cc:3037] T 160e94ea1df443f8bd7a00ac220953c0 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:38.638051 8084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "160e94ea1df443f8bd7a00ac220953c0" candidate_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edba86e2b7a74ab880b1f716db2c7471" is_pre_election: true
I20260619 16:49:38.638157 8084 raft_consensus.cc:2454] T 160e94ea1df443f8bd7a00ac220953c0 P edba86e2b7a74ab880b1f716db2c7471 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f1a774a25df46e18fe29d62a00b8bb9 in term 0.
I20260619 16:49:38.639093 8444 raft_consensus.cc:504] T 160e94ea1df443f8bd7a00ac220953c0 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } }
I20260619 16:49:38.639339 8444 leader_election.cc:288] T 160e94ea1df443f8bd7a00ac220953c0 P 1f1a774a25df46e18fe29d62a00b8bb9 [CANDIDATE]: Term 1 election: Requested vote from peers edba86e2b7a74ab880b1f716db2c7471 (127.0.98.1:37025), 0c052d73c57e481eaa3699c11d96116d (127.0.98.2:38897)
I20260619 16:49:38.639550 8084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "160e94ea1df443f8bd7a00ac220953c0" candidate_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edba86e2b7a74ab880b1f716db2c7471"
I20260619 16:49:38.639618 8224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "160e94ea1df443f8bd7a00ac220953c0" candidate_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c052d73c57e481eaa3699c11d96116d"
I20260619 16:49:38.639690 8084 raft_consensus.cc:3037] T 160e94ea1df443f8bd7a00ac220953c0 P edba86e2b7a74ab880b1f716db2c7471 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:38.639700 8224 raft_consensus.cc:3037] T 160e94ea1df443f8bd7a00ac220953c0 P 0c052d73c57e481eaa3699c11d96116d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:38.640416 8084 raft_consensus.cc:2454] T 160e94ea1df443f8bd7a00ac220953c0 P edba86e2b7a74ab880b1f716db2c7471 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f1a774a25df46e18fe29d62a00b8bb9 in term 1.
I20260619 16:49:38.640489 8224 raft_consensus.cc:2454] T 160e94ea1df443f8bd7a00ac220953c0 P 0c052d73c57e481eaa3699c11d96116d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f1a774a25df46e18fe29d62a00b8bb9 in term 1.
I20260619 16:49:38.640618 8297 leader_election.cc:302] T 160e94ea1df443f8bd7a00ac220953c0 P 1f1a774a25df46e18fe29d62a00b8bb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f1a774a25df46e18fe29d62a00b8bb9, edba86e2b7a74ab880b1f716db2c7471; no voters:
I20260619 16:49:38.640740 8444 raft_consensus.cc:2781] T 160e94ea1df443f8bd7a00ac220953c0 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:38.640926 8444 raft_consensus.cc:686] T 160e94ea1df443f8bd7a00ac220953c0 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 1 LEADER]: Becoming Leader. State: Replica: 1f1a774a25df46e18fe29d62a00b8bb9, State: Running, Role: LEADER
I20260619 16:49:38.641010 8444 consensus_queue.cc:237] T 160e94ea1df443f8bd7a00ac220953c0 P 1f1a774a25df46e18fe29d62a00b8bb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } }
I20260619 16:49:38.641921 7925 catalog_manager.cc:5495] T 160e94ea1df443f8bd7a00ac220953c0 P 1f1a774a25df46e18fe29d62a00b8bb9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f1a774a25df46e18fe29d62a00b8bb9 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } health_report { overall_health: HEALTHY } } }
W20260619 16:49:38.659565 8271 tablet.cc:2369] T 160e94ea1df443f8bd7a00ac220953c0 P 0c052d73c57e481eaa3699c11d96116d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 16:49:38 dist-test-slave-jsw9 krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887777, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 16:49:38.728060 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:38.719384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54458 (local address 127.0.98.62:37881)
0619 16:49:38.719590 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:38.719592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:38.719602 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:49:38.719631 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:38.719632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:38.719642 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:38.719801 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:38.720471 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.721043 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.721872 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.722082 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.722150 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:38.724946 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:38.724947 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:38.724948 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:38.724957 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:38.726504 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:38.727091 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:38.727092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:38.727093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:38.727168 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:38.727416 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:38.727417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:38.727418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:38.727569 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:38.727647 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:38.727825 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 16:49:38.727915 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":105,"threads_started":1}
I20260619 16:49:38.728215 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:38.719241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37881 (local address 127.0.0.1:54458)
0619 16:49:38.719469 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:38.719484 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:38.719566 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:38.719887 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:38.719888 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:38.720294 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:38.720371 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:38.720374 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.721149 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.721150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:38.721752 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:38.721754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.722268 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.722270 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:38.722332 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:38.722986 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:38.722991 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:38.724819 (+ 1828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:38.726609 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:38.726620 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:38.726623 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:38.726990 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:38.727262 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:38.727263 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:38.727263 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:38.727329 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:38.727667 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:38.727675 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:38.727732 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 16:49:38.727800 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 16:49:38.728118 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":96,"thread_start_us":106,"threads_started":1}
I20260619 16:49:38.730556 7925 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54458:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 16:49:38.730813 7925 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:49:38.732800 8064 tablet_service.cc:1467] Processing CreateTablet for tablet 83da56f794a74b0cbd17e69ecca884e0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c5c24e11a0da4ea1bf002c8f46e9e892]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:49:38.732806 8204 tablet_service.cc:1467] Processing CreateTablet for tablet 83da56f794a74b0cbd17e69ecca884e0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c5c24e11a0da4ea1bf002c8f46e9e892]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:49:38.732918 8344 tablet_service.cc:1467] Processing CreateTablet for tablet 83da56f794a74b0cbd17e69ecca884e0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c5c24e11a0da4ea1bf002c8f46e9e892]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:49:38.732987 8064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83da56f794a74b0cbd17e69ecca884e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:38.732991 8204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83da56f794a74b0cbd17e69ecca884e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:38.733048 8344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83da56f794a74b0cbd17e69ecca884e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:38.734534 8438 tablet_bootstrap.cc:492] T 83da56f794a74b0cbd17e69ecca884e0 P edba86e2b7a74ab880b1f716db2c7471: Bootstrap starting.
I20260619 16:49:38.734778 8439 tablet_bootstrap.cc:492] T 83da56f794a74b0cbd17e69ecca884e0 P 1f1a774a25df46e18fe29d62a00b8bb9: Bootstrap starting.
I20260619 16:49:38.734870 8440 tablet_bootstrap.cc:492] T 83da56f794a74b0cbd17e69ecca884e0 P 0c052d73c57e481eaa3699c11d96116d: Bootstrap starting.
I20260619 16:49:38.735090 8438 tablet_bootstrap.cc:654] T 83da56f794a74b0cbd17e69ecca884e0 P edba86e2b7a74ab880b1f716db2c7471: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:38.735244 8439 tablet_bootstrap.cc:654] T 83da56f794a74b0cbd17e69ecca884e0 P 1f1a774a25df46e18fe29d62a00b8bb9: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:38.735258 8440 tablet_bootstrap.cc:654] T 83da56f794a74b0cbd17e69ecca884e0 P 0c052d73c57e481eaa3699c11d96116d: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:38.735903 8440 tablet_bootstrap.cc:492] T 83da56f794a74b0cbd17e69ecca884e0 P 0c052d73c57e481eaa3699c11d96116d: No bootstrap required, opened a new log
I20260619 16:49:38.735927 8439 tablet_bootstrap.cc:492] T 83da56f794a74b0cbd17e69ecca884e0 P 1f1a774a25df46e18fe29d62a00b8bb9: No bootstrap required, opened a new log
I20260619 16:49:38.735958 8440 ts_tablet_manager.cc:1397] T 83da56f794a74b0cbd17e69ecca884e0 P 0c052d73c57e481eaa3699c11d96116d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:38.735975 8439 ts_tablet_manager.cc:1397] T 83da56f794a74b0cbd17e69ecca884e0 P 1f1a774a25df46e18fe29d62a00b8bb9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260619 16:49:38.736006 8438 tablet_bootstrap.cc:492] T 83da56f794a74b0cbd17e69ecca884e0 P edba86e2b7a74ab880b1f716db2c7471: No bootstrap required, opened a new log
I20260619 16:49:38.736079 8438 ts_tablet_manager.cc:1397] T 83da56f794a74b0cbd17e69ecca884e0 P edba86e2b7a74ab880b1f716db2c7471: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:49:38.736138 8440 raft_consensus.cc:348] T 83da56f794a74b0cbd17e69ecca884e0 P 0c052d73c57e481eaa3699c11d96116d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } }
I20260619 16:49:38.736220 8440 raft_consensus.cc:374] T 83da56f794a74b0cbd17e69ecca884e0 P 0c052d73c57e481eaa3699c11d96116d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:38.736243 8440 raft_consensus.cc:729] T 83da56f794a74b0cbd17e69ecca884e0 P 0c052d73c57e481eaa3699c11d96116d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c052d73c57e481eaa3699c11d96116d, State: Initialized, Role: FOLLOWER
I20260619 16:49:38.736284 8440 consensus_queue.cc:260] T 83da56f794a74b0cbd17e69ecca884e0 P 0c052d73c57e481eaa3699c11d96116d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } }
I20260619 16:49:38.736405 8438 raft_consensus.cc:348] T 83da56f794a74b0cbd17e69ecca884e0 P edba86e2b7a74ab880b1f716db2c7471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } }
I20260619 16:49:38.736459 8440 ts_tablet_manager.cc:1428] T 83da56f794a74b0cbd17e69ecca884e0 P 0c052d73c57e481eaa3699c11d96116d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 16:49:38.736305 8439 raft_consensus.cc:348] T 83da56f794a74b0cbd17e69ecca884e0 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } }
I20260619 16:49:38.736510 8438 raft_consensus.cc:374] T 83da56f794a74b0cbd17e69ecca884e0 P edba86e2b7a74ab880b1f716db2c7471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:38.736517 8439 raft_consensus.cc:374] T 83da56f794a74b0cbd17e69ecca884e0 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:38.736536 8438 raft_consensus.cc:729] T 83da56f794a74b0cbd17e69ecca884e0 P edba86e2b7a74ab880b1f716db2c7471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edba86e2b7a74ab880b1f716db2c7471, State: Initialized, Role: FOLLOWER
I20260619 16:49:38.736541 8439 raft_consensus.cc:729] T 83da56f794a74b0cbd17e69ecca884e0 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f1a774a25df46e18fe29d62a00b8bb9, State: Initialized, Role: FOLLOWER
I20260619 16:49:38.736642 8439 consensus_queue.cc:260] T 83da56f794a74b0cbd17e69ecca884e0 P 1f1a774a25df46e18fe29d62a00b8bb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } }
I20260619 16:49:38.736644 8438 consensus_queue.cc:260] T 83da56f794a74b0cbd17e69ecca884e0 P edba86e2b7a74ab880b1f716db2c7471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } }
I20260619 16:49:38.736793 8438 ts_tablet_manager.cc:1428] T 83da56f794a74b0cbd17e69ecca884e0 P edba86e2b7a74ab880b1f716db2c7471: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:38.736793 8439 ts_tablet_manager.cc:1428] T 83da56f794a74b0cbd17e69ecca884e0 P 1f1a774a25df46e18fe29d62a00b8bb9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 16:49:38.736691 8445 tablet_replica.cc:439] T 83da56f794a74b0cbd17e69ecca884e0 P 0c052d73c57e481eaa3699c11d96116d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } } }
I20260619 16:49:38.736902 8445 tablet_replica.cc:442] T 83da56f794a74b0cbd17e69ecca884e0 P 0c052d73c57e481eaa3699c11d96116d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:38.736948 8444 tablet_replica.cc:439] T 83da56f794a74b0cbd17e69ecca884e0 P 1f1a774a25df46e18fe29d62a00b8bb9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } } }
I20260619 16:49:38.736946 8446 tablet_replica.cc:439] T 83da56f794a74b0cbd17e69ecca884e0 P edba86e2b7a74ab880b1f716db2c7471: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } } }
I20260619 16:49:38.737021 8446 tablet_replica.cc:442] T 83da56f794a74b0cbd17e69ecca884e0 P edba86e2b7a74ab880b1f716db2c7471: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:38.737021 8444 tablet_replica.cc:442] T 83da56f794a74b0cbd17e69ecca884e0 P 1f1a774a25df46e18fe29d62a00b8bb9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:38.754609 8446 raft_consensus.cc:482] T 83da56f794a74b0cbd17e69ecca884e0 P edba86e2b7a74ab880b1f716db2c7471 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:38.754755 8446 raft_consensus.cc:504] T 83da56f794a74b0cbd17e69ecca884e0 P edba86e2b7a74ab880b1f716db2c7471 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } }
I20260619 16:49:38.755159 8446 leader_election.cc:288] T 83da56f794a74b0cbd17e69ecca884e0 P edba86e2b7a74ab880b1f716db2c7471 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c052d73c57e481eaa3699c11d96116d (127.0.98.2:38897), 1f1a774a25df46e18fe29d62a00b8bb9 (127.0.98.3:40539)
I20260619 16:49:38.759541 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:38.755647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60773 (local address 127.0.98.3:40539)
0619 16:49:38.755721 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:38.755724 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:38.755779 (+ 55us) server_negotiation.cc:407] Connection header received
0619 16:49:38.755879 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:38.755880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:38.755888 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:38.755938 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:38.756099 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.756648 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.758109 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.758720 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.758824 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:38.758843 (+ 19us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:38.759203 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 16:49:38.759451 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 16:49:38.759580 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:38.755568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43959 (local address 127.0.98.2:38897)
0619 16:49:38.755637 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:38.755640 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:38.755784 (+ 144us) server_negotiation.cc:407] Connection header received
0619 16:49:38.755863 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:38.755864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:38.755882 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:38.755932 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:49:38.756181 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.756670 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.758630 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.759182 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.759255 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:38.759274 (+ 19us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:38.759436 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 16:49:38.759495 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 16:49:38.759713 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:38.755400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38897 (local address 127.0.98.1:43959)
0619 16:49:38.755682 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:38.755695 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:38.755786 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:38.755940 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:38.755941 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:38.755944 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:38.756063 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:38.756065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.756742 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.756743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:38.758442 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:38.758453 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.759274 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.759275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:38.759363 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:38.759367 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:49:38.759416 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 16:49:38.759482 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":108,"thread_start_us":172,"threads_started":1}
I20260619 16:49:38.760360 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:38.755459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40539 (local address 127.0.98.1:60773)
0619 16:49:38.755714 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:38.755725 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:38.755797 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:38.755951 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:38.755952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:38.755954 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:49:38.756032 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:38.756035 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.756735 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.756737 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:38.757913 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:38.757918 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.758897 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.758898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:38.758994 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:38.759000 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 16:49:38.759176 (+ 176us) client_negotiation.cc:240] Negotiation successful
0619 16:49:38.759429 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":135,"thread_start_us":118,"threads_started":1}
I20260619 16:49:38.761174 8364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83da56f794a74b0cbd17e69ecca884e0" candidate_uuid: "edba86e2b7a74ab880b1f716db2c7471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" is_pre_election: true
I20260619 16:49:38.761216 8224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83da56f794a74b0cbd17e69ecca884e0" candidate_uuid: "edba86e2b7a74ab880b1f716db2c7471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c052d73c57e481eaa3699c11d96116d" is_pre_election: true
I20260619 16:49:38.761337 8364 raft_consensus.cc:2454] T 83da56f794a74b0cbd17e69ecca884e0 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edba86e2b7a74ab880b1f716db2c7471 in term 0.
I20260619 16:49:38.761337 8224 raft_consensus.cc:2454] T 83da56f794a74b0cbd17e69ecca884e0 P 0c052d73c57e481eaa3699c11d96116d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edba86e2b7a74ab880b1f716db2c7471 in term 0.
I20260619 16:49:38.761713 8017 leader_election.cc:302] T 83da56f794a74b0cbd17e69ecca884e0 P edba86e2b7a74ab880b1f716db2c7471 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c052d73c57e481eaa3699c11d96116d, edba86e2b7a74ab880b1f716db2c7471; no voters:
I20260619 16:49:38.761904 8446 raft_consensus.cc:2781] T 83da56f794a74b0cbd17e69ecca884e0 P edba86e2b7a74ab880b1f716db2c7471 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:49:38.761984 8446 raft_consensus.cc:482] T 83da56f794a74b0cbd17e69ecca884e0 P edba86e2b7a74ab880b1f716db2c7471 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:49:38.762007 8446 raft_consensus.cc:3037] T 83da56f794a74b0cbd17e69ecca884e0 P edba86e2b7a74ab880b1f716db2c7471 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:38.762931 8446 raft_consensus.cc:504] T 83da56f794a74b0cbd17e69ecca884e0 P edba86e2b7a74ab880b1f716db2c7471 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } }
I20260619 16:49:38.763106 8446 leader_election.cc:288] T 83da56f794a74b0cbd17e69ecca884e0 P edba86e2b7a74ab880b1f716db2c7471 [CANDIDATE]: Term 1 election: Requested vote from peers 0c052d73c57e481eaa3699c11d96116d (127.0.98.2:38897), 1f1a774a25df46e18fe29d62a00b8bb9 (127.0.98.3:40539)
I20260619 16:49:38.763300 8364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83da56f794a74b0cbd17e69ecca884e0" candidate_uuid: "edba86e2b7a74ab880b1f716db2c7471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f1a774a25df46e18fe29d62a00b8bb9"
I20260619 16:49:38.763414 8364 raft_consensus.cc:3037] T 83da56f794a74b0cbd17e69ecca884e0 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:38.764295 8364 raft_consensus.cc:2454] T 83da56f794a74b0cbd17e69ecca884e0 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edba86e2b7a74ab880b1f716db2c7471 in term 1.
I20260619 16:49:38.764489 8018 leader_election.cc:302] T 83da56f794a74b0cbd17e69ecca884e0 P edba86e2b7a74ab880b1f716db2c7471 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f1a774a25df46e18fe29d62a00b8bb9, edba86e2b7a74ab880b1f716db2c7471; no voters:
I20260619 16:49:38.764513 8224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83da56f794a74b0cbd17e69ecca884e0" candidate_uuid: "edba86e2b7a74ab880b1f716db2c7471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c052d73c57e481eaa3699c11d96116d"
I20260619 16:49:38.764591 8446 raft_consensus.cc:2781] T 83da56f794a74b0cbd17e69ecca884e0 P edba86e2b7a74ab880b1f716db2c7471 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:38.764616 8224 raft_consensus.cc:3037] T 83da56f794a74b0cbd17e69ecca884e0 P 0c052d73c57e481eaa3699c11d96116d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:38.764815 8446 raft_consensus.cc:686] T 83da56f794a74b0cbd17e69ecca884e0 P edba86e2b7a74ab880b1f716db2c7471 [term 1 LEADER]: Becoming Leader. State: Replica: edba86e2b7a74ab880b1f716db2c7471, State: Running, Role: LEADER
I20260619 16:49:38.764909 8446 consensus_queue.cc:237] T 83da56f794a74b0cbd17e69ecca884e0 P edba86e2b7a74ab880b1f716db2c7471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } }
I20260619 16:49:38.765484 8482 tablet_replica.cc:439] T 83da56f794a74b0cbd17e69ecca884e0 P edba86e2b7a74ab880b1f716db2c7471: TxnStatusTablet state changed. Reason: New leader edba86e2b7a74ab880b1f716db2c7471. Latest consensus state: current_term: 1 leader_uuid: "edba86e2b7a74ab880b1f716db2c7471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } } }
I20260619 16:49:38.765625 8482 tablet_replica.cc:442] T 83da56f794a74b0cbd17e69ecca884e0 P edba86e2b7a74ab880b1f716db2c7471: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:49:38.765647 8224 raft_consensus.cc:2454] T 83da56f794a74b0cbd17e69ecca884e0 P 0c052d73c57e481eaa3699c11d96116d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edba86e2b7a74ab880b1f716db2c7471 in term 1.
I20260619 16:49:38.765793 7926 catalog_manager.cc:5495] T 83da56f794a74b0cbd17e69ecca884e0 P edba86e2b7a74ab880b1f716db2c7471 reported cstate change: term changed from 0 to 1, leader changed from <none> to edba86e2b7a74ab880b1f716db2c7471 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "edba86e2b7a74ab880b1f716db2c7471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } health_report { overall_health: UNKNOWN } } }
I20260619 16:49:38.766116 8484 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 16:49:38.766295 8484 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 16:49:38.779079 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:38.775939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37025 (local address 127.0.0.1:60102)
0619 16:49:38.776019 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:38.776035 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:38.776097 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:38.776289 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:38.776289 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:38.776291 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:49:38.776355 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:38.776358 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.777378 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.777379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:38.778029 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:38.778032 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.778538 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.778539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:38.778583 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:38.778604 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:49:38.778920 (+ 316us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:49:38.778925 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:49:38.778952 (+ 27us) client_negotiation.cc:240] Negotiation successful
0619 16:49:38.779009 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260619 16:49:38.779088 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:38.776038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60102 (local address 127.0.98.1:37025)
0619 16:49:38.776095 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:38.776098 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:38.776110 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:49:38.776181 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:38.776182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:38.776206 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:38.776285 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:38.776423 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.777289 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.778111 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.778470 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.778542 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:38.778698 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:38.778864 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:38.778945 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:38.778962 (+ 17us) server_negotiation.cc:299] Negotiation successful
0619 16:49:38.778986 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 16:49:38.780483 8364 raft_consensus.cc:1261] T 83da56f794a74b0cbd17e69ecca884e0 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 1 FOLLOWER]: Refusing update from remote peer edba86e2b7a74ab880b1f716db2c7471: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:38.780527 8224 raft_consensus.cc:1261] T 83da56f794a74b0cbd17e69ecca884e0 P 0c052d73c57e481eaa3699c11d96116d [term 1 FOLLOWER]: Refusing update from remote peer edba86e2b7a74ab880b1f716db2c7471: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:38.780854 8482 consensus_queue.cc:1035] T 83da56f794a74b0cbd17e69ecca884e0 P edba86e2b7a74ab880b1f716db2c7471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:38.781008 8446 consensus_queue.cc:1035] T 83da56f794a74b0cbd17e69ecca884e0 P edba86e2b7a74ab880b1f716db2c7471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:38.781723 8444 tablet_replica.cc:439] T 83da56f794a74b0cbd17e69ecca884e0 P 1f1a774a25df46e18fe29d62a00b8bb9: TxnStatusTablet state changed. Reason: New leader edba86e2b7a74ab880b1f716db2c7471. Latest consensus state: current_term: 1 leader_uuid: "edba86e2b7a74ab880b1f716db2c7471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } } }
I20260619 16:49:38.781831 8444 tablet_replica.cc:442] T 83da56f794a74b0cbd17e69ecca884e0 P 1f1a774a25df46e18fe29d62a00b8bb9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:38.781894 8445 tablet_replica.cc:439] T 83da56f794a74b0cbd17e69ecca884e0 P 0c052d73c57e481eaa3699c11d96116d: TxnStatusTablet state changed. Reason: New leader edba86e2b7a74ab880b1f716db2c7471. Latest consensus state: current_term: 1 leader_uuid: "edba86e2b7a74ab880b1f716db2c7471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } } }
I20260619 16:49:38.781989 8445 tablet_replica.cc:442] T 83da56f794a74b0cbd17e69ecca884e0 P 0c052d73c57e481eaa3699c11d96116d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:38.782872 8446 tablet_replica.cc:439] T 83da56f794a74b0cbd17e69ecca884e0 P edba86e2b7a74ab880b1f716db2c7471: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "edba86e2b7a74ab880b1f716db2c7471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } } }
I20260619 16:49:38.782974 8446 tablet_replica.cc:442] T 83da56f794a74b0cbd17e69ecca884e0 P edba86e2b7a74ab880b1f716db2c7471: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:49:38.783510 8451 mvcc.cc:205] Tried to move back new op lower bound from 7298612341882470400 to 7298612341821853696. Current Snapshot: MvccSnapshot[applied={T|T < 7298612341882470400}]
I20260619 16:49:38.783624 8445 tablet_replica.cc:439] T 83da56f794a74b0cbd17e69ecca884e0 P 0c052d73c57e481eaa3699c11d96116d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "edba86e2b7a74ab880b1f716db2c7471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } } }
I20260619 16:49:38.783761 8445 tablet_replica.cc:442] T 83da56f794a74b0cbd17e69ecca884e0 P 0c052d73c57e481eaa3699c11d96116d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:38.783713 8482 tablet_replica.cc:439] T 83da56f794a74b0cbd17e69ecca884e0 P edba86e2b7a74ab880b1f716db2c7471: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "edba86e2b7a74ab880b1f716db2c7471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } } }
I20260619 16:49:38.783847 8482 tablet_replica.cc:442] T 83da56f794a74b0cbd17e69ecca884e0 P edba86e2b7a74ab880b1f716db2c7471: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:49:38.784832 8444 tablet_replica.cc:439] T 83da56f794a74b0cbd17e69ecca884e0 P 1f1a774a25df46e18fe29d62a00b8bb9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "edba86e2b7a74ab880b1f716db2c7471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } } }
I20260619 16:49:38.784955 8444 tablet_replica.cc:442] T 83da56f794a74b0cbd17e69ecca884e0 P 1f1a774a25df46e18fe29d62a00b8bb9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:49:38.787787 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:38.784277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45050 (local address 127.0.98.3:40539)
0619 16:49:38.784376 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:38.784378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:38.784385 (+ 7us) server_negotiation.cc:407] Connection header received
0619 16:49:38.784427 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:38.784428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:38.784450 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:38.784528 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:38.785089 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.785923 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.786792 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.787114 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.787184 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:38.787302 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:38.787468 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:38.787545 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:38.787658 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 16:49:38.787690 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 16:49:38.794237 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:38.791072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59209 (local address 127.0.98.1:37025)
0619 16:49:38.791129 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:38.791131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:38.791228 (+ 97us) server_negotiation.cc:407] Connection header received
0619 16:49:38.791283 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:38.791284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:38.791297 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:38.791357 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:38.791544 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.792411 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.793278 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.793592 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.793675 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:38.793878 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:38.793921 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:38.793977 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:38.794061 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 16:49:38.794090 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 16:49:38.794307 8492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:38.790991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37025 (local address 127.0.98.3:59209)
0619 16:49:38.791156 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:38.791168 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:38.791235 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:38.791411 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:38.791412 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:38.791413 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:49:38.791487 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:38.791489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.792521 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.792522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:38.793181 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:38.793184 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.793690 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.793691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:38.793757 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:38.793780 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:49:38.794002 (+ 222us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:49:38.794005 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:49:38.794057 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 16:49:38.794147 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
I20260619 16:49:38.798125 8224 raft_consensus.cc:1261] T 160e94ea1df443f8bd7a00ac220953c0 P 0c052d73c57e481eaa3699c11d96116d [term 1 FOLLOWER]: Refusing update from remote peer 1f1a774a25df46e18fe29d62a00b8bb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:38.798095 8084 raft_consensus.cc:1261] T 160e94ea1df443f8bd7a00ac220953c0 P edba86e2b7a74ab880b1f716db2c7471 [term 1 FOLLOWER]: Refusing update from remote peer 1f1a774a25df46e18fe29d62a00b8bb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:38.798496 8468 consensus_queue.cc:1035] T 160e94ea1df443f8bd7a00ac220953c0 P 1f1a774a25df46e18fe29d62a00b8bb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:38.798727 8444 consensus_queue.cc:1035] T 160e94ea1df443f8bd7a00ac220953c0 P 1f1a774a25df46e18fe29d62a00b8bb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:38.800696 8452 mvcc.cc:205] Tried to move back new op lower bound from 7298612341952987136 to 7298612341314551808. Current Snapshot: MvccSnapshot[applied={T|T < 7298612341952987136}]
I20260619 16:49:38.814033 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:38.810796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33481 (local address 127.0.98.3:40539)
0619 16:49:38.810848 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:38.810850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:38.811091 (+ 241us) server_negotiation.cc:407] Connection header received
0619 16:49:38.811164 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:38.811165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:38.811178 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:38.811240 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:38.811422 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.811945 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.813305 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.813526 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.813585 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:38.813768 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:38.813809 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:38.813852 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:38.813901 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 16:49:38.813930 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 16:49:38.814066 8496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:38.810731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40539 (local address 127.0.98.1:33481)
0619 16:49:38.811016 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:38.811030 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:38.811099 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:38.811246 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:38.811247 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:38.811249 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:49:38.811365 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:38.811367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.812049 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.812050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:38.813221 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:38.813225 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.813591 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.813592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:38.813666 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:38.813691 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:49:38.813857 (+ 166us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:49:38.813859 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 16:49:38.813900 (+ 41us) client_negotiation.cc:240] Negotiation successful
0619 16:49:38.813953 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":137,"threads_started":1}
I20260619 16:49:38.826164 7926 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:38.826365 7926 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:38.827380 7926 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 384f080274ce451a8180fbc2fbda74ec: Sending DeleteTablet for 3 replicas of tablet 160e94ea1df443f8bd7a00ac220953c0
I20260619 16:49:38.827816 8344 tablet_service.cc:1514] Processing DeleteTablet for tablet 160e94ea1df443f8bd7a00ac220953c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:38 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45036
I20260619 16:49:38.827832 8204 tablet_service.cc:1514] Processing DeleteTablet for tablet 160e94ea1df443f8bd7a00ac220953c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:38 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48358
I20260619 16:49:38.827966 8461 tablet_replica.cc:330] T 160e94ea1df443f8bd7a00ac220953c0 P 0c052d73c57e481eaa3699c11d96116d: stopping tablet replica
I20260619 16:49:38.827963 8462 tablet_replica.cc:330] T 160e94ea1df443f8bd7a00ac220953c0 P 1f1a774a25df46e18fe29d62a00b8bb9: stopping tablet replica
I20260619 16:49:38.828032 8461 raft_consensus.cc:2229] T 160e94ea1df443f8bd7a00ac220953c0 P 0c052d73c57e481eaa3699c11d96116d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:38.828042 8462 raft_consensus.cc:2229] T 160e94ea1df443f8bd7a00ac220953c0 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:49:38.828083 8461 raft_consensus.cc:2258] T 160e94ea1df443f8bd7a00ac220953c0 P 0c052d73c57e481eaa3699c11d96116d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:38.828181 8462 raft_consensus.cc:2258] T 160e94ea1df443f8bd7a00ac220953c0 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:38.828598 8462 ts_tablet_manager.cc:1905] T 160e94ea1df443f8bd7a00ac220953c0 P 1f1a774a25df46e18fe29d62a00b8bb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:38.828657 8461 ts_tablet_manager.cc:1905] T 160e94ea1df443f8bd7a00ac220953c0 P 0c052d73c57e481eaa3699c11d96116d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:38.830821 8462 ts_tablet_manager.cc:1918] T 160e94ea1df443f8bd7a00ac220953c0 P 1f1a774a25df46e18fe29d62a00b8bb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:49:38.830902 8462 log.cc:1196] T 160e94ea1df443f8bd7a00ac220953c0 P 1f1a774a25df46e18fe29d62a00b8bb9: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-2/wal/wals/160e94ea1df443f8bd7a00ac220953c0
I20260619 16:49:38.831146 8462 ts_tablet_manager.cc:1939] T 160e94ea1df443f8bd7a00ac220953c0 P 1f1a774a25df46e18fe29d62a00b8bb9: Deleting consensus metadata
I20260619 16:49:38.831506 8064 tablet_service.cc:1514] Processing DeleteTablet for tablet 160e94ea1df443f8bd7a00ac220953c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:38 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60088
I20260619 16:49:38.831661 8460 tablet_replica.cc:330] T 160e94ea1df443f8bd7a00ac220953c0 P edba86e2b7a74ab880b1f716db2c7471: stopping tablet replica
I20260619 16:49:38.831727 8460 raft_consensus.cc:2229] T 160e94ea1df443f8bd7a00ac220953c0 P edba86e2b7a74ab880b1f716db2c7471 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:38.831780 8460 raft_consensus.cc:2258] T 160e94ea1df443f8bd7a00ac220953c0 P edba86e2b7a74ab880b1f716db2c7471 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:38.832232 8460 ts_tablet_manager.cc:1905] T 160e94ea1df443f8bd7a00ac220953c0 P edba86e2b7a74ab880b1f716db2c7471: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:38.834246 8460 ts_tablet_manager.cc:1918] T 160e94ea1df443f8bd7a00ac220953c0 P edba86e2b7a74ab880b1f716db2c7471: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:49:38.834311 8460 log.cc:1196] T 160e94ea1df443f8bd7a00ac220953c0 P edba86e2b7a74ab880b1f716db2c7471: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-0/wal/wals/160e94ea1df443f8bd7a00ac220953c0
I20260619 16:49:38.834529 8460 ts_tablet_manager.cc:1939] T 160e94ea1df443f8bd7a00ac220953c0 P edba86e2b7a74ab880b1f716db2c7471: Deleting consensus metadata
I20260619 16:49:38.834806 8461 ts_tablet_manager.cc:1918] T 160e94ea1df443f8bd7a00ac220953c0 P 0c052d73c57e481eaa3699c11d96116d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:49:38.834874 8461 log.cc:1196] T 160e94ea1df443f8bd7a00ac220953c0 P 0c052d73c57e481eaa3699c11d96116d: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-1/wal/wals/160e94ea1df443f8bd7a00ac220953c0
I20260619 16:49:38.835078 8461 ts_tablet_manager.cc:1939] T 160e94ea1df443f8bd7a00ac220953c0 P 0c052d73c57e481eaa3699c11d96116d: Deleting consensus metadata
I20260619 16:49:38.835552 7913 catalog_manager.cc:4841] TS 0c052d73c57e481eaa3699c11d96116d (127.0.98.2:38897): tablet 160e94ea1df443f8bd7a00ac220953c0 (table test-table [id=2cbcd2a91e094e4ebf2b992b3e6da07d]) successfully deleted
I20260619 16:49:38.835923 7913 catalog_manager.cc:4841] TS edba86e2b7a74ab880b1f716db2c7471 (127.0.98.1:37025): tablet 160e94ea1df443f8bd7a00ac220953c0 (table test-table [id=2cbcd2a91e094e4ebf2b992b3e6da07d]) successfully deleted
I20260619 16:49:38.836298 7914 catalog_manager.cc:4841] TS 1f1a774a25df46e18fe29d62a00b8bb9 (127.0.98.3:40539): tablet 160e94ea1df443f8bd7a00ac220953c0 (table test-table [id=2cbcd2a91e094e4ebf2b992b3e6da07d]) successfully deleted
Jun 19 16:49:38 dist-test-slave-jsw9 krb5kdc[7882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 16:49:38 dist-test-slave-jsw9 krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:38 dist-test-slave-jsw9 krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 16:49:38.868798 8476 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:38.859423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54466 (local address 127.0.98.62:37881)
0619 16:49:38.859493 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:38.859495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:38.859641 (+ 146us) server_negotiation.cc:407] Connection header received
0619 16:49:38.859739 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:38.859740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:38.859754 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:38.859888 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:38.860748 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.861630 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.862889 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.863253 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.863352 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:38.868642 (+ 5290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54466: BlockingRecv error: recv got EOF from 127.0.0.1:54466 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Jun 19 16:49:38 dist-test-slave-jsw9 krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 16:49:38.885592 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:38.873493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54474 (local address 127.0.98.62:37881)
0619 16:49:38.873563 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:38.873565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:38.873835 (+ 270us) server_negotiation.cc:407] Connection header received
0619 16:49:38.873905 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:38.873907 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:38.873921 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:38.873977 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:38.875288 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.876196 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.877540 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:38.877890 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:38.878062 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:38.881428 (+ 3366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:38.881431 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:38.881432 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:38.881444 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:38.883957 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:38.884546 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:38.884548 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:38.884549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:38.884632 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:38.884862 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:38.884864 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:38.884864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:38.885053 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:38.885169 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:38.885350 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 16:49:38.885465 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 16:49:38.887981 7926 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:49:38.888229 7926 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:49:38.890048 8204 tablet_service.cc:1467] Processing CreateTablet for tablet c2388ddb9519401dad6082f608bb8938 (DEFAULT_TABLE table=test-table [id=28dea8ce6ce94eb99db77cf5ca39d784]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:38.890048 8064 tablet_service.cc:1467] Processing CreateTablet for tablet c2388ddb9519401dad6082f608bb8938 (DEFAULT_TABLE table=test-table [id=28dea8ce6ce94eb99db77cf5ca39d784]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:38.890273 8064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2388ddb9519401dad6082f608bb8938. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:38.890300 8204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2388ddb9519401dad6082f608bb8938. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:38.891887 8438 tablet_bootstrap.cc:492] T c2388ddb9519401dad6082f608bb8938 P edba86e2b7a74ab880b1f716db2c7471: Bootstrap starting.
I20260619 16:49:38.891940 8440 tablet_bootstrap.cc:492] T c2388ddb9519401dad6082f608bb8938 P 0c052d73c57e481eaa3699c11d96116d: Bootstrap starting.
I20260619 16:49:38.892373 8438 tablet_bootstrap.cc:654] T c2388ddb9519401dad6082f608bb8938 P edba86e2b7a74ab880b1f716db2c7471: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:38.892375 8440 tablet_bootstrap.cc:654] T c2388ddb9519401dad6082f608bb8938 P 0c052d73c57e481eaa3699c11d96116d: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:38.892855 8344 tablet_service.cc:1467] Processing CreateTablet for tablet c2388ddb9519401dad6082f608bb8938 (DEFAULT_TABLE table=test-table [id=28dea8ce6ce94eb99db77cf5ca39d784]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:38.892994 8344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2388ddb9519401dad6082f608bb8938. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:38.893428 8440 tablet_bootstrap.cc:492] T c2388ddb9519401dad6082f608bb8938 P 0c052d73c57e481eaa3699c11d96116d: No bootstrap required, opened a new log
I20260619 16:49:38.893484 8440 ts_tablet_manager.cc:1397] T c2388ddb9519401dad6082f608bb8938 P 0c052d73c57e481eaa3699c11d96116d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:49:38.893623 8440 raft_consensus.cc:348] T c2388ddb9519401dad6082f608bb8938 P 0c052d73c57e481eaa3699c11d96116d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } }
I20260619 16:49:38.893715 8440 raft_consensus.cc:374] T c2388ddb9519401dad6082f608bb8938 P 0c052d73c57e481eaa3699c11d96116d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:38.893733 8440 raft_consensus.cc:729] T c2388ddb9519401dad6082f608bb8938 P 0c052d73c57e481eaa3699c11d96116d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c052d73c57e481eaa3699c11d96116d, State: Initialized, Role: FOLLOWER
I20260619 16:49:38.893782 8440 consensus_queue.cc:260] T c2388ddb9519401dad6082f608bb8938 P 0c052d73c57e481eaa3699c11d96116d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } }
I20260619 16:49:38.893878 8440 ts_tablet_manager.cc:1428] T c2388ddb9519401dad6082f608bb8938 P 0c052d73c57e481eaa3699c11d96116d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:38.894793 8439 tablet_bootstrap.cc:492] T c2388ddb9519401dad6082f608bb8938 P 1f1a774a25df46e18fe29d62a00b8bb9: Bootstrap starting.
I20260619 16:49:38.895404 8439 tablet_bootstrap.cc:654] T c2388ddb9519401dad6082f608bb8938 P 1f1a774a25df46e18fe29d62a00b8bb9: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:38.896348 8438 tablet_bootstrap.cc:492] T c2388ddb9519401dad6082f608bb8938 P edba86e2b7a74ab880b1f716db2c7471: No bootstrap required, opened a new log
I20260619 16:49:38.896422 8438 ts_tablet_manager.cc:1397] T c2388ddb9519401dad6082f608bb8938 P edba86e2b7a74ab880b1f716db2c7471: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260619 16:49:38.896735 8438 raft_consensus.cc:348] T c2388ddb9519401dad6082f608bb8938 P edba86e2b7a74ab880b1f716db2c7471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } }
I20260619 16:49:38.896862 8438 raft_consensus.cc:374] T c2388ddb9519401dad6082f608bb8938 P edba86e2b7a74ab880b1f716db2c7471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:38.896998 8438 raft_consensus.cc:729] T c2388ddb9519401dad6082f608bb8938 P edba86e2b7a74ab880b1f716db2c7471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edba86e2b7a74ab880b1f716db2c7471, State: Initialized, Role: FOLLOWER
I20260619 16:49:38.897109 8438 consensus_queue.cc:260] T c2388ddb9519401dad6082f608bb8938 P edba86e2b7a74ab880b1f716db2c7471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } }
I20260619 16:49:38.897289 8438 ts_tablet_manager.cc:1428] T c2388ddb9519401dad6082f608bb8938 P edba86e2b7a74ab880b1f716db2c7471: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:38.898723 8439 tablet_bootstrap.cc:492] T c2388ddb9519401dad6082f608bb8938 P 1f1a774a25df46e18fe29d62a00b8bb9: No bootstrap required, opened a new log
I20260619 16:49:38.898819 8439 ts_tablet_manager.cc:1397] T c2388ddb9519401dad6082f608bb8938 P 1f1a774a25df46e18fe29d62a00b8bb9: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260619 16:49:38.899051 8439 raft_consensus.cc:348] T c2388ddb9519401dad6082f608bb8938 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } }
I20260619 16:49:38.899145 8439 raft_consensus.cc:374] T c2388ddb9519401dad6082f608bb8938 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:38.899164 8439 raft_consensus.cc:729] T c2388ddb9519401dad6082f608bb8938 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f1a774a25df46e18fe29d62a00b8bb9, State: Initialized, Role: FOLLOWER
I20260619 16:49:38.899216 8439 consensus_queue.cc:260] T c2388ddb9519401dad6082f608bb8938 P 1f1a774a25df46e18fe29d62a00b8bb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } }
I20260619 16:49:38.899322 8439 ts_tablet_manager.cc:1428] T c2388ddb9519401dad6082f608bb8938 P 1f1a774a25df46e18fe29d62a00b8bb9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:39.165887 8468 raft_consensus.cc:482] T c2388ddb9519401dad6082f608bb8938 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:39.166036 8468 raft_consensus.cc:504] T c2388ddb9519401dad6082f608bb8938 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } }
I20260619 16:49:39.166299 8468 leader_election.cc:288] T c2388ddb9519401dad6082f608bb8938 P 1f1a774a25df46e18fe29d62a00b8bb9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edba86e2b7a74ab880b1f716db2c7471 (127.0.98.1:37025), 0c052d73c57e481eaa3699c11d96116d (127.0.98.2:38897)
I20260619 16:49:39.166628 8084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2388ddb9519401dad6082f608bb8938" candidate_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edba86e2b7a74ab880b1f716db2c7471" is_pre_election: true
I20260619 16:49:39.166695 8224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2388ddb9519401dad6082f608bb8938" candidate_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c052d73c57e481eaa3699c11d96116d" is_pre_election: true
I20260619 16:49:39.166778 8084 raft_consensus.cc:2454] T c2388ddb9519401dad6082f608bb8938 P edba86e2b7a74ab880b1f716db2c7471 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f1a774a25df46e18fe29d62a00b8bb9 in term 0.
I20260619 16:49:39.166816 8224 raft_consensus.cc:2454] T c2388ddb9519401dad6082f608bb8938 P 0c052d73c57e481eaa3699c11d96116d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f1a774a25df46e18fe29d62a00b8bb9 in term 0.
I20260619 16:49:39.166949 8297 leader_election.cc:302] T c2388ddb9519401dad6082f608bb8938 P 1f1a774a25df46e18fe29d62a00b8bb9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f1a774a25df46e18fe29d62a00b8bb9, edba86e2b7a74ab880b1f716db2c7471; no voters:
I20260619 16:49:39.167055 8468 raft_consensus.cc:2781] T c2388ddb9519401dad6082f608bb8938 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:49:39.167105 8468 raft_consensus.cc:482] T c2388ddb9519401dad6082f608bb8938 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:49:39.167173 8468 raft_consensus.cc:3037] T c2388ddb9519401dad6082f608bb8938 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:39.168124 8468 raft_consensus.cc:504] T c2388ddb9519401dad6082f608bb8938 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } }
I20260619 16:49:39.168265 8468 leader_election.cc:288] T c2388ddb9519401dad6082f608bb8938 P 1f1a774a25df46e18fe29d62a00b8bb9 [CANDIDATE]: Term 1 election: Requested vote from peers edba86e2b7a74ab880b1f716db2c7471 (127.0.98.1:37025), 0c052d73c57e481eaa3699c11d96116d (127.0.98.2:38897)
I20260619 16:49:39.168491 8084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2388ddb9519401dad6082f608bb8938" candidate_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edba86e2b7a74ab880b1f716db2c7471"
I20260619 16:49:39.168516 8224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2388ddb9519401dad6082f608bb8938" candidate_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c052d73c57e481eaa3699c11d96116d"
I20260619 16:49:39.168577 8084 raft_consensus.cc:3037] T c2388ddb9519401dad6082f608bb8938 P edba86e2b7a74ab880b1f716db2c7471 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:39.168592 8224 raft_consensus.cc:3037] T c2388ddb9519401dad6082f608bb8938 P 0c052d73c57e481eaa3699c11d96116d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:39.169283 8224 raft_consensus.cc:2454] T c2388ddb9519401dad6082f608bb8938 P 0c052d73c57e481eaa3699c11d96116d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f1a774a25df46e18fe29d62a00b8bb9 in term 1.
I20260619 16:49:39.169302 8084 raft_consensus.cc:2454] T c2388ddb9519401dad6082f608bb8938 P edba86e2b7a74ab880b1f716db2c7471 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f1a774a25df46e18fe29d62a00b8bb9 in term 1.
I20260619 16:49:39.169472 8297 leader_election.cc:302] T c2388ddb9519401dad6082f608bb8938 P 1f1a774a25df46e18fe29d62a00b8bb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f1a774a25df46e18fe29d62a00b8bb9, edba86e2b7a74ab880b1f716db2c7471; no voters:
I20260619 16:49:39.169582 8468 raft_consensus.cc:2781] T c2388ddb9519401dad6082f608bb8938 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:39.169649 8468 raft_consensus.cc:686] T c2388ddb9519401dad6082f608bb8938 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 1 LEADER]: Becoming Leader. State: Replica: 1f1a774a25df46e18fe29d62a00b8bb9, State: Running, Role: LEADER
I20260619 16:49:39.169710 8468 consensus_queue.cc:237] T c2388ddb9519401dad6082f608bb8938 P 1f1a774a25df46e18fe29d62a00b8bb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } }
I20260619 16:49:39.170382 7927 catalog_manager.cc:5495] T c2388ddb9519401dad6082f608bb8938 P 1f1a774a25df46e18fe29d62a00b8bb9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f1a774a25df46e18fe29d62a00b8bb9 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } health_report { overall_health: HEALTHY } } }
I20260619 16:49:39.189482 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:39.187108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45060 (local address 127.0.98.3:40539)
0619 16:49:39.187177 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:39.187179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:39.187189 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:49:39.187253 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:39.187254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:39.187266 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:39.187311 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:49:39.187473 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:39.188021 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:39.188812 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:39.189013 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:39.189071 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:39.189221 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:49:39.189258 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:49:39.189299 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:39.189357 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 16:49:39.189384 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 16:49:39.190582 8084 raft_consensus.cc:1261] T c2388ddb9519401dad6082f608bb8938 P edba86e2b7a74ab880b1f716db2c7471 [term 1 FOLLOWER]: Refusing update from remote peer 1f1a774a25df46e18fe29d62a00b8bb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:39.190582 8224 raft_consensus.cc:1261] T c2388ddb9519401dad6082f608bb8938 P 0c052d73c57e481eaa3699c11d96116d [term 1 FOLLOWER]: Refusing update from remote peer 1f1a774a25df46e18fe29d62a00b8bb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:39.190852 8468 consensus_queue.cc:1035] T c2388ddb9519401dad6082f608bb8938 P 1f1a774a25df46e18fe29d62a00b8bb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:39.190986 8444 consensus_queue.cc:1035] T c2388ddb9519401dad6082f608bb8938 P 1f1a774a25df46e18fe29d62a00b8bb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:39.194164 7926 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:39.194331 7926 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:39.194994 7926 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 384f080274ce451a8180fbc2fbda74ec: Sending DeleteTablet for 3 replicas of tablet c2388ddb9519401dad6082f608bb8938
I20260619 16:49:39.195242 8064 tablet_service.cc:1514] Processing DeleteTablet for tablet c2388ddb9519401dad6082f608bb8938 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:39 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60088
I20260619 16:49:39.195292 8344 tablet_service.cc:1514] Processing DeleteTablet for tablet c2388ddb9519401dad6082f608bb8938 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:39 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45036
I20260619 16:49:39.195282 8204 tablet_service.cc:1514] Processing DeleteTablet for tablet c2388ddb9519401dad6082f608bb8938 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:39 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48358
I20260619 16:49:39.195364 8460 tablet_replica.cc:330] T c2388ddb9519401dad6082f608bb8938 P edba86e2b7a74ab880b1f716db2c7471: stopping tablet replica
I20260619 16:49:39.195377 8462 tablet_replica.cc:330] T c2388ddb9519401dad6082f608bb8938 P 1f1a774a25df46e18fe29d62a00b8bb9: stopping tablet replica
I20260619 16:49:39.195381 8461 tablet_replica.cc:330] T c2388ddb9519401dad6082f608bb8938 P 0c052d73c57e481eaa3699c11d96116d: stopping tablet replica
I20260619 16:49:39.195420 8460 raft_consensus.cc:2229] T c2388ddb9519401dad6082f608bb8938 P edba86e2b7a74ab880b1f716db2c7471 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:39.195430 8462 raft_consensus.cc:2229] T c2388ddb9519401dad6082f608bb8938 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:49:39.195436 8461 raft_consensus.cc:2229] T c2388ddb9519401dad6082f608bb8938 P 0c052d73c57e481eaa3699c11d96116d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:39.195482 8460 raft_consensus.cc:2258] T c2388ddb9519401dad6082f608bb8938 P edba86e2b7a74ab880b1f716db2c7471 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:39.195483 8461 raft_consensus.cc:2258] T c2388ddb9519401dad6082f608bb8938 P 0c052d73c57e481eaa3699c11d96116d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:39.195500 8462 raft_consensus.cc:2258] T c2388ddb9519401dad6082f608bb8938 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:39.195865 8461 ts_tablet_manager.cc:1905] T c2388ddb9519401dad6082f608bb8938 P 0c052d73c57e481eaa3699c11d96116d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:39.195961 8460 ts_tablet_manager.cc:1905] T c2388ddb9519401dad6082f608bb8938 P edba86e2b7a74ab880b1f716db2c7471: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:39.197516 7926 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:49:39.197806 7926 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:49:39.197847 8460 ts_tablet_manager.cc:1918] T c2388ddb9519401dad6082f608bb8938 P edba86e2b7a74ab880b1f716db2c7471: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:39.197922 8460 log.cc:1196] T c2388ddb9519401dad6082f608bb8938 P edba86e2b7a74ab880b1f716db2c7471: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-0/wal/wals/c2388ddb9519401dad6082f608bb8938
I20260619 16:49:39.197919 8461 ts_tablet_manager.cc:1918] T c2388ddb9519401dad6082f608bb8938 P 0c052d73c57e481eaa3699c11d96116d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:39.197974 8461 log.cc:1196] T c2388ddb9519401dad6082f608bb8938 P 0c052d73c57e481eaa3699c11d96116d: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-1/wal/wals/c2388ddb9519401dad6082f608bb8938
I20260619 16:49:39.198153 8460 ts_tablet_manager.cc:1939] T c2388ddb9519401dad6082f608bb8938 P edba86e2b7a74ab880b1f716db2c7471: Deleting consensus metadata
I20260619 16:49:39.198230 8461 ts_tablet_manager.cc:1939] T c2388ddb9519401dad6082f608bb8938 P 0c052d73c57e481eaa3699c11d96116d: Deleting consensus metadata
I20260619 16:49:39.198539 8462 ts_tablet_manager.cc:1905] T c2388ddb9519401dad6082f608bb8938 P 1f1a774a25df46e18fe29d62a00b8bb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:39.198877 7913 catalog_manager.cc:4841] TS edba86e2b7a74ab880b1f716db2c7471 (127.0.98.1:37025): tablet c2388ddb9519401dad6082f608bb8938 (table test-table [id=28dea8ce6ce94eb99db77cf5ca39d784]) successfully deleted
I20260619 16:49:39.199470 8204 tablet_service.cc:1467] Processing CreateTablet for tablet e641b4ebce494d1990ba1e04dc224ac9 (DEFAULT_TABLE table=test-table [id=6a6e51a9e4ee497fa35f56f7f6f7e6ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:39.199617 8204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e641b4ebce494d1990ba1e04dc224ac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:39.200865 8462 ts_tablet_manager.cc:1918] T c2388ddb9519401dad6082f608bb8938 P 1f1a774a25df46e18fe29d62a00b8bb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:49:39.200959 8462 log.cc:1196] T c2388ddb9519401dad6082f608bb8938 P 1f1a774a25df46e18fe29d62a00b8bb9: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-2/wal/wals/c2388ddb9519401dad6082f608bb8938
I20260619 16:49:39.201228 8462 ts_tablet_manager.cc:1939] T c2388ddb9519401dad6082f608bb8938 P 1f1a774a25df46e18fe29d62a00b8bb9: Deleting consensus metadata
I20260619 16:49:39.201447 8344 tablet_service.cc:1467] Processing CreateTablet for tablet e641b4ebce494d1990ba1e04dc224ac9 (DEFAULT_TABLE table=test-table [id=6a6e51a9e4ee497fa35f56f7f6f7e6ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:39.201572 8344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e641b4ebce494d1990ba1e04dc224ac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:39.201802 8440 tablet_bootstrap.cc:492] T e641b4ebce494d1990ba1e04dc224ac9 P 0c052d73c57e481eaa3699c11d96116d: Bootstrap starting.
I20260619 16:49:39.201828 7914 catalog_manager.cc:4841] TS 1f1a774a25df46e18fe29d62a00b8bb9 (127.0.98.3:40539): tablet c2388ddb9519401dad6082f608bb8938 (table test-table [id=28dea8ce6ce94eb99db77cf5ca39d784]) successfully deleted
I20260619 16:49:39.202215 8440 tablet_bootstrap.cc:654] T e641b4ebce494d1990ba1e04dc224ac9 P 0c052d73c57e481eaa3699c11d96116d: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:39.202673 7913 catalog_manager.cc:4841] TS 0c052d73c57e481eaa3699c11d96116d (127.0.98.2:38897): tablet c2388ddb9519401dad6082f608bb8938 (table test-table [id=28dea8ce6ce94eb99db77cf5ca39d784]) successfully deleted
I20260619 16:49:39.202708 8064 tablet_service.cc:1467] Processing CreateTablet for tablet e641b4ebce494d1990ba1e04dc224ac9 (DEFAULT_TABLE table=test-table [id=6a6e51a9e4ee497fa35f56f7f6f7e6ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:49:39.202854 8064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e641b4ebce494d1990ba1e04dc224ac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:39.203027 8439 tablet_bootstrap.cc:492] T e641b4ebce494d1990ba1e04dc224ac9 P 1f1a774a25df46e18fe29d62a00b8bb9: Bootstrap starting.
I20260619 16:49:39.203368 8439 tablet_bootstrap.cc:654] T e641b4ebce494d1990ba1e04dc224ac9 P 1f1a774a25df46e18fe29d62a00b8bb9: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:39.203997 8439 tablet_bootstrap.cc:492] T e641b4ebce494d1990ba1e04dc224ac9 P 1f1a774a25df46e18fe29d62a00b8bb9: No bootstrap required, opened a new log
I20260619 16:49:39.204046 8439 ts_tablet_manager.cc:1397] T e641b4ebce494d1990ba1e04dc224ac9 P 1f1a774a25df46e18fe29d62a00b8bb9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260619 16:49:39.204186 8439 raft_consensus.cc:348] T e641b4ebce494d1990ba1e04dc224ac9 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } }
I20260619 16:49:39.204248 8438 tablet_bootstrap.cc:492] T e641b4ebce494d1990ba1e04dc224ac9 P edba86e2b7a74ab880b1f716db2c7471: Bootstrap starting.
I20260619 16:49:39.204268 8439 raft_consensus.cc:374] T e641b4ebce494d1990ba1e04dc224ac9 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:39.204289 8439 raft_consensus.cc:729] T e641b4ebce494d1990ba1e04dc224ac9 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f1a774a25df46e18fe29d62a00b8bb9, State: Initialized, Role: FOLLOWER
I20260619 16:49:39.204327 8439 consensus_queue.cc:260] T e641b4ebce494d1990ba1e04dc224ac9 P 1f1a774a25df46e18fe29d62a00b8bb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } }
I20260619 16:49:39.204402 8440 tablet_bootstrap.cc:492] T e641b4ebce494d1990ba1e04dc224ac9 P 0c052d73c57e481eaa3699c11d96116d: No bootstrap required, opened a new log
I20260619 16:49:39.204408 8444 raft_consensus.cc:482] T e641b4ebce494d1990ba1e04dc224ac9 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:49:39.204440 8439 ts_tablet_manager.cc:1428] T e641b4ebce494d1990ba1e04dc224ac9 P 1f1a774a25df46e18fe29d62a00b8bb9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:39.204450 8440 ts_tablet_manager.cc:1397] T e641b4ebce494d1990ba1e04dc224ac9 P 0c052d73c57e481eaa3699c11d96116d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 16:49:39.204452 8444 raft_consensus.cc:504] T e641b4ebce494d1990ba1e04dc224ac9 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } }
I20260619 16:49:39.204581 8444 leader_election.cc:288] T e641b4ebce494d1990ba1e04dc224ac9 P 1f1a774a25df46e18fe29d62a00b8bb9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edba86e2b7a74ab880b1f716db2c7471 (127.0.98.1:37025), 0c052d73c57e481eaa3699c11d96116d (127.0.98.2:38897)
I20260619 16:49:39.204595 8438 tablet_bootstrap.cc:654] T e641b4ebce494d1990ba1e04dc224ac9 P edba86e2b7a74ab880b1f716db2c7471: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:39.204578 8440 raft_consensus.cc:348] T e641b4ebce494d1990ba1e04dc224ac9 P 0c052d73c57e481eaa3699c11d96116d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } }
I20260619 16:49:39.204655 8440 raft_consensus.cc:374] T e641b4ebce494d1990ba1e04dc224ac9 P 0c052d73c57e481eaa3699c11d96116d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:39.204687 8440 raft_consensus.cc:729] T e641b4ebce494d1990ba1e04dc224ac9 P 0c052d73c57e481eaa3699c11d96116d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c052d73c57e481eaa3699c11d96116d, State: Initialized, Role: FOLLOWER
I20260619 16:49:39.204725 8440 consensus_queue.cc:260] T e641b4ebce494d1990ba1e04dc224ac9 P 0c052d73c57e481eaa3699c11d96116d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } }
I20260619 16:49:39.204819 8440 ts_tablet_manager.cc:1428] T e641b4ebce494d1990ba1e04dc224ac9 P 0c052d73c57e481eaa3699c11d96116d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 16:49:39.204803 8084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e641b4ebce494d1990ba1e04dc224ac9" candidate_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edba86e2b7a74ab880b1f716db2c7471" is_pre_election: true
W20260619 16:49:39.205183 8297 leader_election.cc:341] T e641b4ebce494d1990ba1e04dc224ac9 P 1f1a774a25df46e18fe29d62a00b8bb9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer edba86e2b7a74ab880b1f716db2c7471 (127.0.98.1:37025): Illegal state: must be running to vote when last-logged opid is not known
I20260619 16:49:39.205235 8438 tablet_bootstrap.cc:492] T e641b4ebce494d1990ba1e04dc224ac9 P edba86e2b7a74ab880b1f716db2c7471: No bootstrap required, opened a new log
I20260619 16:49:39.205211 8224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e641b4ebce494d1990ba1e04dc224ac9" candidate_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c052d73c57e481eaa3699c11d96116d" is_pre_election: true
I20260619 16:49:39.205276 8438 ts_tablet_manager.cc:1397] T e641b4ebce494d1990ba1e04dc224ac9 P edba86e2b7a74ab880b1f716db2c7471: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:39.205277 8224 raft_consensus.cc:2454] T e641b4ebce494d1990ba1e04dc224ac9 P 0c052d73c57e481eaa3699c11d96116d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f1a774a25df46e18fe29d62a00b8bb9 in term 0.
I20260619 16:49:39.205422 8297 leader_election.cc:302] T e641b4ebce494d1990ba1e04dc224ac9 P 1f1a774a25df46e18fe29d62a00b8bb9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0c052d73c57e481eaa3699c11d96116d, 1f1a774a25df46e18fe29d62a00b8bb9; no voters: edba86e2b7a74ab880b1f716db2c7471
I20260619 16:49:39.205401 8438 raft_consensus.cc:348] T e641b4ebce494d1990ba1e04dc224ac9 P edba86e2b7a74ab880b1f716db2c7471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } }
I20260619 16:49:39.205462 8438 raft_consensus.cc:374] T e641b4ebce494d1990ba1e04dc224ac9 P edba86e2b7a74ab880b1f716db2c7471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:39.205480 8438 raft_consensus.cc:729] T e641b4ebce494d1990ba1e04dc224ac9 P edba86e2b7a74ab880b1f716db2c7471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edba86e2b7a74ab880b1f716db2c7471, State: Initialized, Role: FOLLOWER
I20260619 16:49:39.205521 8438 consensus_queue.cc:260] T e641b4ebce494d1990ba1e04dc224ac9 P edba86e2b7a74ab880b1f716db2c7471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } }
I20260619 16:49:39.205534 8444 raft_consensus.cc:2781] T e641b4ebce494d1990ba1e04dc224ac9 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:49:39.205564 8444 raft_consensus.cc:482] T e641b4ebce494d1990ba1e04dc224ac9 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:49:39.205581 8444 raft_consensus.cc:3037] T e641b4ebce494d1990ba1e04dc224ac9 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:39.205613 8438 ts_tablet_manager.cc:1428] T e641b4ebce494d1990ba1e04dc224ac9 P edba86e2b7a74ab880b1f716db2c7471: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:39.206343 8444 raft_consensus.cc:504] T e641b4ebce494d1990ba1e04dc224ac9 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } }
I20260619 16:49:39.206465 8444 leader_election.cc:288] T e641b4ebce494d1990ba1e04dc224ac9 P 1f1a774a25df46e18fe29d62a00b8bb9 [CANDIDATE]: Term 1 election: Requested vote from peers edba86e2b7a74ab880b1f716db2c7471 (127.0.98.1:37025), 0c052d73c57e481eaa3699c11d96116d (127.0.98.2:38897)
I20260619 16:49:39.206615 8084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e641b4ebce494d1990ba1e04dc224ac9" candidate_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edba86e2b7a74ab880b1f716db2c7471"
I20260619 16:49:39.206657 8224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e641b4ebce494d1990ba1e04dc224ac9" candidate_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c052d73c57e481eaa3699c11d96116d"
I20260619 16:49:39.206699 8084 raft_consensus.cc:3037] T e641b4ebce494d1990ba1e04dc224ac9 P edba86e2b7a74ab880b1f716db2c7471 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:39.206723 8224 raft_consensus.cc:3037] T e641b4ebce494d1990ba1e04dc224ac9 P 0c052d73c57e481eaa3699c11d96116d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:39.207365 8224 raft_consensus.cc:2454] T e641b4ebce494d1990ba1e04dc224ac9 P 0c052d73c57e481eaa3699c11d96116d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f1a774a25df46e18fe29d62a00b8bb9 in term 1.
I20260619 16:49:39.207505 8084 raft_consensus.cc:2454] T e641b4ebce494d1990ba1e04dc224ac9 P edba86e2b7a74ab880b1f716db2c7471 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f1a774a25df46e18fe29d62a00b8bb9 in term 1.
I20260619 16:49:39.207535 8297 leader_election.cc:302] T e641b4ebce494d1990ba1e04dc224ac9 P 1f1a774a25df46e18fe29d62a00b8bb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c052d73c57e481eaa3699c11d96116d, 1f1a774a25df46e18fe29d62a00b8bb9; no voters:
I20260619 16:49:39.207619 8444 raft_consensus.cc:2781] T e641b4ebce494d1990ba1e04dc224ac9 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:39.207682 8444 raft_consensus.cc:686] T e641b4ebce494d1990ba1e04dc224ac9 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 1 LEADER]: Becoming Leader. State: Replica: 1f1a774a25df46e18fe29d62a00b8bb9, State: Running, Role: LEADER
I20260619 16:49:39.207737 8444 consensus_queue.cc:237] T e641b4ebce494d1990ba1e04dc224ac9 P 1f1a774a25df46e18fe29d62a00b8bb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } }
I20260619 16:49:39.208740 7926 catalog_manager.cc:5495] T e641b4ebce494d1990ba1e04dc224ac9 P 1f1a774a25df46e18fe29d62a00b8bb9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f1a774a25df46e18fe29d62a00b8bb9 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f1a774a25df46e18fe29d62a00b8bb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40539 } health_report { overall_health: HEALTHY } } }
I20260619 16:49:39.218439 8224 raft_consensus.cc:1261] T e641b4ebce494d1990ba1e04dc224ac9 P 0c052d73c57e481eaa3699c11d96116d [term 1 FOLLOWER]: Refusing update from remote peer 1f1a774a25df46e18fe29d62a00b8bb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:39.218483 8084 raft_consensus.cc:1261] T e641b4ebce494d1990ba1e04dc224ac9 P edba86e2b7a74ab880b1f716db2c7471 [term 1 FOLLOWER]: Refusing update from remote peer 1f1a774a25df46e18fe29d62a00b8bb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:49:39.218679 8444 consensus_queue.cc:1035] T e641b4ebce494d1990ba1e04dc224ac9 P 1f1a774a25df46e18fe29d62a00b8bb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edba86e2b7a74ab880b1f716db2c7471" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:39.218824 8468 consensus_queue.cc:1035] T e641b4ebce494d1990ba1e04dc224ac9 P 1f1a774a25df46e18fe29d62a00b8bb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c052d73c57e481eaa3699c11d96116d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:49:39.229791 7926 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:39.229939 7926 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:49:39.230702 7926 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 384f080274ce451a8180fbc2fbda74ec: Sending DeleteTablet for 3 replicas of tablet e641b4ebce494d1990ba1e04dc224ac9
I20260619 16:49:39.230980 8064 tablet_service.cc:1514] Processing DeleteTablet for tablet e641b4ebce494d1990ba1e04dc224ac9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:39 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60088
I20260619 16:49:39.231015 8204 tablet_service.cc:1514] Processing DeleteTablet for tablet e641b4ebce494d1990ba1e04dc224ac9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:39 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48358
I20260619 16:49:39.231093 8460 tablet_replica.cc:330] T e641b4ebce494d1990ba1e04dc224ac9 P edba86e2b7a74ab880b1f716db2c7471: stopping tablet replica
I20260619 16:49:39.231063 8344 tablet_service.cc:1514] Processing DeleteTablet for tablet e641b4ebce494d1990ba1e04dc224ac9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:49:39 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45036
I20260619 16:49:39.231166 8461 tablet_replica.cc:330] T e641b4ebce494d1990ba1e04dc224ac9 P 0c052d73c57e481eaa3699c11d96116d: stopping tablet replica
I20260619 16:49:39.231195 8460 raft_consensus.cc:2229] T e641b4ebce494d1990ba1e04dc224ac9 P edba86e2b7a74ab880b1f716db2c7471 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:39.231240 8460 raft_consensus.cc:2258] T e641b4ebce494d1990ba1e04dc224ac9 P edba86e2b7a74ab880b1f716db2c7471 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:39.231238 8461 raft_consensus.cc:2229] T e641b4ebce494d1990ba1e04dc224ac9 P 0c052d73c57e481eaa3699c11d96116d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:49:39.231281 8461 raft_consensus.cc:2258] T e641b4ebce494d1990ba1e04dc224ac9 P 0c052d73c57e481eaa3699c11d96116d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:39.231354 8462 tablet_replica.cc:330] T e641b4ebce494d1990ba1e04dc224ac9 P 1f1a774a25df46e18fe29d62a00b8bb9: stopping tablet replica
I20260619 16:49:39.231364 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 7999
I20260619 16:49:39.231413 8462 raft_consensus.cc:2229] T e641b4ebce494d1990ba1e04dc224ac9 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:49:39.231477 8462 raft_consensus.cc:2258] T e641b4ebce494d1990ba1e04dc224ac9 P 1f1a774a25df46e18fe29d62a00b8bb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:49:39.232429 8461 ts_tablet_manager.cc:1905] T e641b4ebce494d1990ba1e04dc224ac9 P 0c052d73c57e481eaa3699c11d96116d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:39.234369 8461 ts_tablet_manager.cc:1918] T e641b4ebce494d1990ba1e04dc224ac9 P 0c052d73c57e481eaa3699c11d96116d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:49:39.234452 8461 log.cc:1196] T e641b4ebce494d1990ba1e04dc224ac9 P 0c052d73c57e481eaa3699c11d96116d: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-1/wal/wals/e641b4ebce494d1990ba1e04dc224ac9
I20260619 16:49:39.234777 8461 ts_tablet_manager.cc:1939] T e641b4ebce494d1990ba1e04dc224ac9 P 0c052d73c57e481eaa3699c11d96116d: Deleting consensus metadata
I20260619 16:49:39.235464 7913 catalog_manager.cc:4841] TS 0c052d73c57e481eaa3699c11d96116d (127.0.98.2:38897): tablet e641b4ebce494d1990ba1e04dc224ac9 (table test-table [id=6a6e51a9e4ee497fa35f56f7f6f7e6ce]) successfully deleted
W20260619 16:49:39.239555 7913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37025 (error 108)
I20260619 16:49:39.239609 8462 ts_tablet_manager.cc:1905] T e641b4ebce494d1990ba1e04dc224ac9 P 1f1a774a25df46e18fe29d62a00b8bb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:49:39.240399 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:39.240088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37025 (local address 127.0.0.1:60106)
0619 16:49:39.240272 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:39.240335 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":69,"threads_started":1}
I20260619 16:49:39.240515 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 8139
W20260619 16:49:39.240581 7913 catalog_manager.cc:4584] TS edba86e2b7a74ab880b1f716db2c7471 (127.0.98.1:37025): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e641b4ebce494d1990ba1e04dc224ac9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37025: connect: Connection refused (error 111)
I20260619 16:49:39.242534 8462 ts_tablet_manager.cc:1918] T e641b4ebce494d1990ba1e04dc224ac9 P 1f1a774a25df46e18fe29d62a00b8bb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:49:39.242621 8462 log.cc:1196] T e641b4ebce494d1990ba1e04dc224ac9 P 1f1a774a25df46e18fe29d62a00b8bb9: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781887741903642-392-0/minicluster-data/ts-2/wal/wals/e641b4ebce494d1990ba1e04dc224ac9
I20260619 16:49:39.243127 8462 ts_tablet_manager.cc:1939] T e641b4ebce494d1990ba1e04dc224ac9 P 1f1a774a25df46e18fe29d62a00b8bb9: Deleting consensus metadata
I20260619 16:49:39.243675 7914 catalog_manager.cc:4841] TS 1f1a774a25df46e18fe29d62a00b8bb9 (127.0.98.3:40539): tablet e641b4ebce494d1990ba1e04dc224ac9 (table test-table [id=6a6e51a9e4ee497fa35f56f7f6f7e6ce]) successfully deleted
I20260619 16:49:39.247092 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 8279
I20260619 16:49:39.253603 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 7895
2026-06-19T16:49:39Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3179 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:49:39 dist-test-slave-jsw9 krb5kdc[8529](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:49:39 dist-test-slave-jsw9 krb5kdc[8529](info): set up 2 sockets
Jun 19 16:49:39 dist-test-slave-jsw9 krb5kdc[8529](info): commencing operation
krb5kdc: starting...
W20260619 16:49:40.276245 392 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:49:40 dist-test-slave-jsw9 krb5kdc[8529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:49:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:49:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:49:40.417214 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33007
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:40.516517 8542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:40.516841 8542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:40.516881 8542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:40.518800 8542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:40.518895 8542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:40.518920 8542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:40.518939 8542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:40.518956 8542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:49:40.521127 8542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:40.521503 8542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:40.521781 8542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:40.528355 8550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:40.528360 8547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:40.528369 8548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:40.528803 8542 server_base.cc:1034] running on GCE node
I20260619 16:49:40.529079 8542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:40.529440 8542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:40.530614 8542 hybrid_clock.cc:648] HybridClock initialized: now 1781887780530578 us; error 56 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T16:49:40Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1291 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:49:40 dist-test-slave-jsw9 krb5kdc[8555](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:49:40 dist-test-slave-jsw9 krb5kdc[8555](info): set up 2 sockets
Jun 19 16:49:40 dist-test-slave-jsw9 krb5kdc[8555](info): commencing operation
krb5kdc: starting...
W20260619 16:49:41.595711 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:49:41 dist-test-slave-jsw9 krb5kdc[8555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:49:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:49:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:49:41.750764 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43111
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:41.855298 8568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:41.855592 8568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:41.855631 8568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:41.858338 8568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:41.858410 8568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:41.858436 8568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:41.858454 8568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:41.858472 8568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:49:41.862979 8568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:41.863368 8568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:41.863642 8568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:41.869927 8573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:41.869910 8576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:41.869910 8574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:41.870303 8568 server_base.cc:1034] running on GCE node
I20260619 16:49:41.870570 8568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:41.870946 8568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:41.872159 8568 hybrid_clock.cc:648] HybridClock initialized: now 1781887781872144 us; error 33 us; skew 500 ppm
Jun 19 16:49:41 dist-test-slave-jsw9 krb5kdc[8555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887781, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:41.875241 8568 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:49:41.875849 8568 webserver.cc:460] Webserver started at http://127.0.98.62:46621/ using document root <none> and password file <none>
I20260619 16:49:41.876088 8568 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:41.876143 8568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:41.876304 8568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:41.877636 8568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "f675a9f7a27f4ef9a92c340d28e0ffb2"
format_stamp: "Formatted at 2026-06-19 16:49:41 on dist-test-slave-jsw9"
server_key: "0d2ddcd025da11d732b96636add0d108"
server_key_iv: "6bece81953d738580daec01f430d7caa"
server_key_version: "encryptionkey@0"
I20260619 16:49:41.878270 8568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "f675a9f7a27f4ef9a92c340d28e0ffb2"
format_stamp: "Formatted at 2026-06-19 16:49:41 on dist-test-slave-jsw9"
server_key: "0d2ddcd025da11d732b96636add0d108"
server_key_iv: "6bece81953d738580daec01f430d7caa"
server_key_version: "encryptionkey@0"
I20260619 16:49:41.879834 8568 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:49:41.880870 8583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:41.881086 8568 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:49:41.881188 8568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "f675a9f7a27f4ef9a92c340d28e0ffb2"
format_stamp: "Formatted at 2026-06-19 16:49:41 on dist-test-slave-jsw9"
server_key: "0d2ddcd025da11d732b96636add0d108"
server_key_iv: "6bece81953d738580daec01f430d7caa"
server_key_version: "encryptionkey@0"
I20260619 16:49:41.881309 8568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:41.892814 8568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:41.895339 8568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:41.895502 8568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:41.900399 8568 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43111
I20260619 16:49:41.900439 8635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43111 every 8 connection(s)
I20260619 16:49:41.901026 8568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:41.901795 8636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:41.904733 8636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2: Bootstrap starting.
I20260619 16:49:41.905493 8636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:41.905812 8636 log.cc:824] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:41.906718 8636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2: No bootstrap required, opened a new log
I20260619 16:49:41.907776 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 8568
I20260619 16:49:41.907887 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:49:41.908078 392 external_mini_cluster.cc:1428] Setting key 2707f6fa0ff03bfd18934c1c87fafb22
I20260619 16:49:41.908742 8636 raft_consensus.cc:348] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f675a9f7a27f4ef9a92c340d28e0ffb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43111 } }
I20260619 16:49:41.908917 8636 raft_consensus.cc:374] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:41.908941 8636 raft_consensus.cc:729] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f675a9f7a27f4ef9a92c340d28e0ffb2, State: Initialized, Role: FOLLOWER
I20260619 16:49:41.909437 8636 consensus_queue.cc:260] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f675a9f7a27f4ef9a92c340d28e0ffb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43111 } }
I20260619 16:49:41.909582 8636 raft_consensus.cc:388] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:41.909617 8636 raft_consensus.cc:482] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:41.909668 8636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:41.910574 8636 raft_consensus.cc:504] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f675a9f7a27f4ef9a92c340d28e0ffb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43111 } }
I20260619 16:49:41.910701 8636 leader_election.cc:302] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f675a9f7a27f4ef9a92c340d28e0ffb2; no voters:
I20260619 16:49:41.910905 8636 leader_election.cc:288] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:49:41.910983 8641 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:41.911201 8636 sys_catalog.cc:564] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:41.911265 8641 raft_consensus.cc:686] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: f675a9f7a27f4ef9a92c340d28e0ffb2, State: Running, Role: LEADER
I20260619 16:49:41.911417 8641 consensus_queue.cc:237] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f675a9f7a27f4ef9a92c340d28e0ffb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43111 } }
I20260619 16:49:41.911998 8643 sys_catalog.cc:455] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f675a9f7a27f4ef9a92c340d28e0ffb2. Latest consensus state: current_term: 1 leader_uuid: "f675a9f7a27f4ef9a92c340d28e0ffb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f675a9f7a27f4ef9a92c340d28e0ffb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43111 } } }
I20260619 16:49:41.911979 8642 sys_catalog.cc:455] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f675a9f7a27f4ef9a92c340d28e0ffb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f675a9f7a27f4ef9a92c340d28e0ffb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43111 } } }
I20260619 16:49:41.912083 8643 sys_catalog.cc:458] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:41.912089 8642 sys_catalog.cc:458] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:41.912343 8650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:41.913024 8650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:41.915006 8650 catalog_manager.cc:1347] Generated new cluster ID: 5678360183c04cbaa694ddf584bdf77e
I20260619 16:49:41.915073 8650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 16:49:41 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:41.922031 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:41.908724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51650 (local address 127.0.98.62:43111)
0619 16:49:41.909481 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:41.909498 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:41.909539 (+ 41us) server_negotiation.cc:407] Connection header received
0619 16:49:41.909681 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:41.909691 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:41.909760 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:41.909935 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:41.910711 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:41.912191 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:41.913469 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:41.914142 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:41.914340 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:41.917556 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:41.917562 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:41.917562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:41.917581 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:41.919869 (+ 2288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:41.920444 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:41.920448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:41.920449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:41.920589 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:41.920801 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:41.920801 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:41.920802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:41.921205 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:41.921386 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:41.921549 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 16:49:41.921733 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":604,"threads_started":1}
I20260619 16:49:41.927510 8650 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:49:41.928346 8650 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:41.931708 8650 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2: Generated new TSK 0
I20260619 16:49:41.931941 8650 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:49:41.971026 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43111
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:42.076566 8662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:42.076856 8662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:42.076889 8662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:42.078459 8662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:42.078655 8662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:42.078786 8662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:49:42.081727 8662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:42.082274 8662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:42.082608 8662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:42.088774 8670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:42.088774 8667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:42.088984 8662 server_base.cc:1034] running on GCE node
W20260619 16:49:42.088775 8668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:42.089321 8662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:42.089648 8662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:42.090852 8662 hybrid_clock.cc:648] HybridClock initialized: now 1781887782090832 us; error 39 us; skew 500 ppm
Jun 19 16:49:42 dist-test-slave-jsw9 krb5kdc[8555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:42.094262 8662 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:49:42.094825 8662 webserver.cc:460] Webserver started at http://127.0.98.1:42943/ using document root <none> and password file <none>
I20260619 16:49:42.095053 8662 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:42.095103 8662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:42.095237 8662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:42.096505 8662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-0/data/instance:
uuid: "be4f31650ff54fae8c5c8ab16cddb64c"
format_stamp: "Formatted at 2026-06-19 16:49:42 on dist-test-slave-jsw9"
server_key: "7db70e0ce0450eae9ecc2b47fb3c5007"
server_key_iv: "f084be2c0c5658358e0ff30197d90677"
server_key_version: "encryptionkey@0"
I20260619 16:49:42.096879 8662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-0/wal/instance:
uuid: "be4f31650ff54fae8c5c8ab16cddb64c"
format_stamp: "Formatted at 2026-06-19 16:49:42 on dist-test-slave-jsw9"
server_key: "7db70e0ce0450eae9ecc2b47fb3c5007"
server_key_iv: "f084be2c0c5658358e0ff30197d90677"
server_key_version: "encryptionkey@0"
I20260619 16:49:42.098726 8662 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:49:42.099797 8677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:42.100021 8662 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:42.100117 8662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-0/wal
uuid: "be4f31650ff54fae8c5c8ab16cddb64c"
format_stamp: "Formatted at 2026-06-19 16:49:42 on dist-test-slave-jsw9"
server_key: "7db70e0ce0450eae9ecc2b47fb3c5007"
server_key_iv: "f084be2c0c5658358e0ff30197d90677"
server_key_version: "encryptionkey@0"
I20260619 16:49:42.100186 8662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:42.118072 8662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:42.120548 8662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:42.120690 8662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:42.120949 8662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:42.121309 8662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:42.121347 8662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:42.121372 8662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:42.121383 8662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:42.128232 8662 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40069
I20260619 16:49:42.128297 8790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40069 every 8 connection(s)
I20260619 16:49:42.128669 8662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 16:49:42 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:42.138526 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 8662
I20260619 16:49:42.138654 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:49:42.138885 392 external_mini_cluster.cc:1428] Setting key 579d2426ca6f2484b4e6016dd1167a2d
I20260619 16:49:42.140206 8793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:42.129480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.1:43205)
0619 16:49:42.129662 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:42.129711 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:42.130772 (+ 1061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:42.131101 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:42.131106 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:42.131437 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:42.131667 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:42.131672 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:42.132506 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:42.132507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:42.133426 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:42.133429 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:42.133851 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:42.133852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:42.133982 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:42.134681 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:42.134695 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:42.136404 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:42.138293 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:42.138324 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:42.138328 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:42.138720 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:42.138967 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:42.138968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:42.138969 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:42.139141 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:42.139416 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:42.139423 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:42.139596 (+ 173us) client_negotiation.cc:769] Sending connection context
0619 16:49:42.139707 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 16:49:42.139919 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":87,"threads_started":1}
I20260619 16:49:42.140604 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:42.129555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43205 (local address 127.0.98.62:43111)
0619 16:49:42.129621 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:42.129623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:42.129827 (+ 204us) server_negotiation.cc:407] Connection header received
0619 16:49:42.130899 (+ 1072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:42.130901 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:42.130918 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:42.130995 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:42.131792 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:42.132412 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:42.133522 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:42.133747 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:42.133814 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:42.136525 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:42.136528 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:42.136528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:42.136536 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:42.138161 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:42.138817 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:42.138818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:42.138819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:42.138898 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:42.139211 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:42.139212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:42.139212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:42.139354 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:42.139417 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:42.140400 (+ 983us) server_negotiation.cc:299] Negotiation successful
0619 16:49:42.140511 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 16:49:42.141006 8791 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43111
I20260619 16:49:42.141135 8791 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:42.141376 8791 heartbeater.cc:511] Master 127.0.98.62:43111 requested a full tablet report, sending...
I20260619 16:49:42.142165 8600 ts_manager.cc:194] Registered new tserver with Master: be4f31650ff54fae8c5c8ab16cddb64c (127.0.98.1:40069)
I20260619 16:49:42.142879 8600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43205
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:49:42.170389 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43111
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:42.267629 8796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:42.267932 8796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:42.267975 8796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:42.269728 8796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:42.269836 8796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:42.269897 8796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:49:42.272183 8796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:42.272541 8796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:42.273546 8796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:42.280313 8801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:42.280455 8802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:42.280369 8804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:42.280362 8796 server_base.cc:1034] running on GCE node
I20260619 16:49:42.280812 8796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:42.281103 8796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:42.282294 8796 hybrid_clock.cc:648] HybridClock initialized: now 1781887782282276 us; error 35 us; skew 500 ppm
Jun 19 16:49:42 dist-test-slave-jsw9 krb5kdc[8555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:42.285279 8796 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:49:42.285848 8796 webserver.cc:460] Webserver started at http://127.0.98.2:45159/ using document root <none> and password file <none>
I20260619 16:49:42.286092 8796 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:42.286144 8796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:42.286309 8796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:42.287621 8796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-1/data/instance:
uuid: "28a05403b261428ba7052d0df43d97c0"
format_stamp: "Formatted at 2026-06-19 16:49:42 on dist-test-slave-jsw9"
server_key: "4c58b86f55df333d01adbdecdcebe1e9"
server_key_iv: "70ca456daadc828e09bfd1185ae466df"
server_key_version: "encryptionkey@0"
I20260619 16:49:42.287999 8796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-1/wal/instance:
uuid: "28a05403b261428ba7052d0df43d97c0"
format_stamp: "Formatted at 2026-06-19 16:49:42 on dist-test-slave-jsw9"
server_key: "4c58b86f55df333d01adbdecdcebe1e9"
server_key_iv: "70ca456daadc828e09bfd1185ae466df"
server_key_version: "encryptionkey@0"
I20260619 16:49:42.289465 8796 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 16:49:42.290702 8811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:42.291061 8796 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:49:42.291221 8796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-1/wal
uuid: "28a05403b261428ba7052d0df43d97c0"
format_stamp: "Formatted at 2026-06-19 16:49:42 on dist-test-slave-jsw9"
server_key: "4c58b86f55df333d01adbdecdcebe1e9"
server_key_iv: "70ca456daadc828e09bfd1185ae466df"
server_key_version: "encryptionkey@0"
I20260619 16:49:42.291306 8796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:42.308666 8796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:42.311447 8796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:42.311632 8796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:42.311928 8796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:42.312340 8796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:42.312376 8796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:42.312409 8796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:42.312439 8796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:42.320273 8796 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41797
I20260619 16:49:42.320312 8924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41797 every 8 connection(s)
I20260619 16:49:42.320819 8796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:49:42.327533 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 8796
I20260619 16:49:42.327667 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:49:42.327883 392 external_mini_cluster.cc:1428] Setting key 667292457ff519172b8797c6f6c1cbc3
Jun 19 16:49:42 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:42.334139 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:42.321851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44997 (local address 127.0.98.62:43111)
0619 16:49:42.322057 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:42.322059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:42.322299 (+ 240us) server_negotiation.cc:407] Connection header received
0619 16:49:42.323365 (+ 1066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:42.323367 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:42.323379 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:42.323443 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:42.324405 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:42.325504 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:42.327047 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:42.327312 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:42.327401 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:42.330773 (+ 3372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:42.330776 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:42.330776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:42.330787 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:42.332237 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:42.332756 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:42.332757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:42.332757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:42.332815 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:42.333119 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:42.333121 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:42.333121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:42.333315 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:42.333404 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:42.333897 (+ 493us) server_negotiation.cc:299] Negotiation successful
0619 16:49:42.334023 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20260619 16:49:42.334239 8927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:42.321823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.2:44997)
0619 16:49:42.322054 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:42.322108 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:42.323221 (+ 1113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:42.323567 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:42.323573 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:42.323936 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:42.324288 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:42.324294 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:42.325653 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:42.325654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:42.326911 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:42.326916 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:42.327430 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:42.327432 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:42.327578 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:42.328319 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:42.328335 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:42.330629 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:42.332345 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:42.332359 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:42.332364 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:42.332654 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:42.332873 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:42.332874 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:42.332874 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:42.333002 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:42.333490 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:42.333497 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:42.333679 (+ 182us) client_negotiation.cc:769] Sending connection context
0619 16:49:42.333787 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 16:49:42.333976 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":113,"threads_started":1}
I20260619 16:49:42.334991 8925 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43111
I20260619 16:49:42.335112 8925 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:42.335628 8925 heartbeater.cc:511] Master 127.0.98.62:43111 requested a full tablet report, sending...
I20260619 16:49:42.336413 8600 ts_manager.cc:194] Registered new tserver with Master: 28a05403b261428ba7052d0df43d97c0 (127.0.98.2:41797)
I20260619 16:49:42.337010 8600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44997
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:49:42.369930 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43111
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:42.469374 8930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:42.469671 8930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:42.469703 8930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:42.471278 8930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:42.471354 8930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:42.471400 8930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:49:42.473327 8930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:42.473737 8930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:42.474035 8930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:42.480939 8936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:42.481027 8930 server_base.cc:1034] running on GCE node
W20260619 16:49:42.481096 8935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:42.481096 8938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:42.481544 8930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:42.481871 8930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:42.483059 8930 hybrid_clock.cc:648] HybridClock initialized: now 1781887782483031 us; error 49 us; skew 500 ppm
Jun 19 16:49:42 dist-test-slave-jsw9 krb5kdc[8555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:42.486364 8930 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:49:42.486953 8930 webserver.cc:460] Webserver started at http://127.0.98.3:32777/ using document root <none> and password file <none>
I20260619 16:49:42.487190 8930 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:42.487241 8930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:42.487397 8930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:42.488833 8930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-2/data/instance:
uuid: "99a6b200fc8b4cac919fff622a4941c9"
format_stamp: "Formatted at 2026-06-19 16:49:42 on dist-test-slave-jsw9"
server_key: "821ceedb2ef805f3c53ab434b60338b0"
server_key_iv: "bb0e3429ce01525f4107000092a78c37"
server_key_version: "encryptionkey@0"
I20260619 16:49:42.489362 8930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-2/wal/instance:
uuid: "99a6b200fc8b4cac919fff622a4941c9"
format_stamp: "Formatted at 2026-06-19 16:49:42 on dist-test-slave-jsw9"
server_key: "821ceedb2ef805f3c53ab434b60338b0"
server_key_iv: "bb0e3429ce01525f4107000092a78c37"
server_key_version: "encryptionkey@0"
I20260619 16:49:42.491030 8930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260619 16:49:42.491998 8945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:42.492338 8930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:42.492445 8930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-2/wal
uuid: "99a6b200fc8b4cac919fff622a4941c9"
format_stamp: "Formatted at 2026-06-19 16:49:42 on dist-test-slave-jsw9"
server_key: "821ceedb2ef805f3c53ab434b60338b0"
server_key_iv: "bb0e3429ce01525f4107000092a78c37"
server_key_version: "encryptionkey@0"
I20260619 16:49:42.492527 8930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:42.507589 8930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:42.510404 8930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:42.510581 8930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:42.510857 8930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:42.511202 8930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:42.511238 8930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:42.511271 8930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:42.511296 8930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:42.518348 8930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33563
I20260619 16:49:42.518431 9058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33563 every 8 connection(s)
I20260619 16:49:42.518813 8930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 16:49:42 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:49:42.527159 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 8930
I20260619 16:49:42.527347 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:49:42.527680 392 external_mini_cluster.cc:1428] Setting key a836c4f104d22fd9ef109e1e9c29129a
I20260619 16:49:42.531216 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:42.519745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56387 (local address 127.0.98.62:43111)
0619 16:49:42.519846 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:42.519848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:42.520095 (+ 247us) server_negotiation.cc:407] Connection header received
0619 16:49:42.521141 (+ 1046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:42.521143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:42.521158 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:42.521260 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:42.522166 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:42.522777 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:42.524157 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:42.524388 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:42.524469 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:42.527810 (+ 3341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:42.527813 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:42.527813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:42.527825 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:42.529426 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:42.529957 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:42.529958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:42.529959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:42.530022 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:42.530456 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:42.530457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:42.530457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:42.530604 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:42.530665 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:42.531046 (+ 381us) server_negotiation.cc:299] Negotiation successful
0619 16:49:42.531134 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:49:42.531468 9061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:42.519698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.3:56387)
0619 16:49:42.519883 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:42.519934 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:42.520981 (+ 1047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:42.521390 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:42.521395 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:42.521726 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:42.521977 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:42.521984 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:42.522971 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:42.522972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:42.523963 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:42.523966 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:42.524506 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:42.524507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:42.524650 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:42.525399 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:42.525409 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:42.527637 (+ 2228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:42.529523 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:42.529534 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:42.529537 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:42.529875 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:42.530124 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:49:42.530125 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:49:42.530125 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:49:42.530381 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:49:42.530684 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:42.530691 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:42.530872 (+ 181us) client_negotiation.cc:769] Sending connection context
0619 16:49:42.530983 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 16:49:42.531155 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":86,"threads_started":1}
I20260619 16:49:42.532039 9059 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43111
I20260619 16:49:42.532155 9059 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:42.532383 9059 heartbeater.cc:511] Master 127.0.98.62:43111 requested a full tablet report, sending...
I20260619 16:49:42.532843 8600 ts_manager.cc:194] Registered new tserver with Master: 99a6b200fc8b4cac919fff622a4941c9 (127.0.98.3:33563)
I20260619 16:49:42.533376 8600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56387
I20260619 16:49:42.540544 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 16:49:42.644260 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 8662
I20260619 16:49:42.649204 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 8796
I20260619 16:49:42.653944 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 8930
I20260619 16:49:42.658509 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 8568
I20260619 16:49:42.664491 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43111
--webserver_interface=127.0.98.62
--webserver_port=46621
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:49:42.765537 9068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:42.765836 9068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:42.765864 9068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:42.768158 9068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:42.768265 9068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:42.768280 9068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:42.768293 9068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:42.768306 9068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:49:42.771405 9068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:42.771867 9068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:42.772150 9068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:42.778342 9074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:42.778317 9076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:42.778288 9073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:42.778542 9068 server_base.cc:1034] running on GCE node
I20260619 16:49:42.779072 9068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:42.779453 9068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:42.780632 9068 hybrid_clock.cc:648] HybridClock initialized: now 1781887782780610 us; error 37 us; skew 500 ppm
Jun 19 16:49:42 dist-test-slave-jsw9 krb5kdc[8555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:42.784149 9068 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:49:42.784859 9068 webserver.cc:460] Webserver started at http://127.0.98.62:46621/ using document root <none> and password file <none>
I20260619 16:49:42.785105 9068 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:42.785182 9068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:42.786862 9068 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:42.787708 9083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:42.787949 9068 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:42.788055 9068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "f675a9f7a27f4ef9a92c340d28e0ffb2"
format_stamp: "Formatted at 2026-06-19 16:49:41 on dist-test-slave-jsw9"
server_key: "0d2ddcd025da11d732b96636add0d108"
server_key_iv: "6bece81953d738580daec01f430d7caa"
server_key_version: "encryptionkey@0"
I20260619 16:49:42.788516 9068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:42.801919 9068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:42.804692 9068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:42.804881 9068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:42.810457 9068 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43111
I20260619 16:49:42.810544 9135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43111 every 8 connection(s)
I20260619 16:49:42.811177 9068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:42.811453 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 9068
I20260619 16:49:42.811784 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40069
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43111
--webserver_port=42943
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:49:42.813114 9136 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:49:42.814154 9136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2: Bootstrap starting.
I20260619 16:49:42.827677 9136 log.cc:824] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:42.830711 9136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:49:42.831137 9136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2: Bootstrap complete.
I20260619 16:49:42.834875 9136 raft_consensus.cc:348] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f675a9f7a27f4ef9a92c340d28e0ffb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43111 } }
I20260619 16:49:42.835314 9136 raft_consensus.cc:729] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f675a9f7a27f4ef9a92c340d28e0ffb2, State: Initialized, Role: FOLLOWER
I20260619 16:49:42.835525 9136 consensus_queue.cc:260] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f675a9f7a27f4ef9a92c340d28e0ffb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43111 } }
I20260619 16:49:42.835597 9136 raft_consensus.cc:388] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:42.835646 9136 raft_consensus.cc:482] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:42.835707 9136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2 [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:49:42.837487 9136 raft_consensus.cc:504] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f675a9f7a27f4ef9a92c340d28e0ffb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43111 } }
I20260619 16:49:42.837670 9136 leader_election.cc:302] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f675a9f7a27f4ef9a92c340d28e0ffb2; no voters:
I20260619 16:49:42.837896 9136 leader_election.cc:288] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 16:49:42.838130 9140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 16:49:42.838301 9136 sys_catalog.cc:564] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:42.838441 9140 raft_consensus.cc:686] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2 [term 2 LEADER]: Becoming Leader. State: Replica: f675a9f7a27f4ef9a92c340d28e0ffb2, State: Running, Role: LEADER
I20260619 16:49:42.838644 9140 consensus_queue.cc:237] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f675a9f7a27f4ef9a92c340d28e0ffb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43111 } }
I20260619 16:49:42.839123 9142 sys_catalog.cc:455] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f675a9f7a27f4ef9a92c340d28e0ffb2. Latest consensus state: current_term: 2 leader_uuid: "f675a9f7a27f4ef9a92c340d28e0ffb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f675a9f7a27f4ef9a92c340d28e0ffb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43111 } } }
I20260619 16:49:42.839233 9142 sys_catalog.cc:458] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:42.839116 9141 sys_catalog.cc:455] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f675a9f7a27f4ef9a92c340d28e0ffb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f675a9f7a27f4ef9a92c340d28e0ffb2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43111 } } }
I20260619 16:49:42.839447 9149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:42.839491 9141 sys_catalog.cc:458] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2 [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:42.840236 9149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:42.840739 9149 catalog_manager.cc:1259] Loaded cluster ID: 5678360183c04cbaa694ddf584bdf77e
I20260619 16:49:42.840801 9149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:49:42.841987 9149 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:42.842470 9149 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f675a9f7a27f4ef9a92c340d28e0ffb2: Loaded TSK: 0
I20260619 16:49:42.843032 9149 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 16:49:42.937456 9138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:42.937768 9138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:42.937812 9138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:42.939844 9138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:42.939922 9138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:42.939972 9138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:49:42.941895 9138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40069
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42943
--tserver_master_addrs=127.0.98.62:43111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:42.942286 9138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:42.942798 9138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:49:42.949900 9138 server_base.cc:1034] running on GCE node
W20260619 16:49:42.949952 9162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:42.950100 9163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:42.950371 9165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:42.950620 9138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:42.950945 9138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:42.952140 9138 hybrid_clock.cc:648] HybridClock initialized: now 1781887782952112 us; error 49 us; skew 500 ppm
Jun 19 16:49:42 dist-test-slave-jsw9 krb5kdc[8555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887782, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:42.955914 9138 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 16:49:42.956764 9138 webserver.cc:460] Webserver started at http://127.0.98.1:42943/ using document root <none> and password file <none>
I20260619 16:49:42.957063 9138 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:42.957144 9138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:42.958796 9138 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 16:49:42.959532 9172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:42.959779 9138 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:49:42.959883 9138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-0/wal
uuid: "be4f31650ff54fae8c5c8ab16cddb64c"
format_stamp: "Formatted at 2026-06-19 16:49:42 on dist-test-slave-jsw9"
server_key: "7db70e0ce0450eae9ecc2b47fb3c5007"
server_key_iv: "f084be2c0c5658358e0ff30197d90677"
server_key_version: "encryptionkey@0"
I20260619 16:49:42.960779 9138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:42.977173 9138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:42.980254 9138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:42.980437 9138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:42.980721 9138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:42.981073 9138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:42.981109 9138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:42.981145 9138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:42.981169 9138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:42.988423 9138 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40069
I20260619 16:49:42.988483 9285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40069 every 8 connection(s)
I20260619 16:49:42.988970 9138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:49:42.989802 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 9138
I20260619 16:49:42.990150 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41797
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43111
--webserver_port=45159
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:49:42 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:42 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:43.001655 9288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:42.989912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.1:42151)
0619 16:49:42.990242 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:42.990292 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:42.991229 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:42.992482 (+ 1253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:42.992489 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:42.993772 (+ 1283us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:42.994139 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:42.994149 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:42.996224 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:42.996226 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:42.997343 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:42.997347 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:42.998097 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:42.998099 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:42.998258 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:42.998877 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:42.998887 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:43.001382 (+ 2495us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":92,"threads_started":1}
W20260619 16:49:43.001791 9289 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:42.990035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42151 (local address 127.0.98.62:43111)
0619 16:49:42.991557 (+ 1522us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:42.991585 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:42.991635 (+ 50us) server_negotiation.cc:407] Connection header received
0619 16:49:42.991797 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:42.991803 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:42.992228 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:42.994363 (+ 2135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:42.994469 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:42.996038 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:42.997472 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:42.997984 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:42.998101 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:43.001581 (+ 3480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42151: BlockingRecv error: recv got EOF from 127.0.98.1:42151 (error 108)
Metrics: {"server-negotiator.queue_time_us":1449,"thread_start_us":96,"threads_started":1}
W20260619 16:49:43.002645 9286 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43111 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:43111: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 19 16:49:43 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:43 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:43.009804 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:43.003397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.1:39965)
0619 16:49:43.003595 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:43.003608 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:43.003693 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:43.004822 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:43.004824 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:43.005030 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:43.005104 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:43.005107 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.005807 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.005808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:43.006476 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:43.006480 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.006848 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.006849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:43.006922 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:43.007449 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:43.007453 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:43.009680 (+ 2227us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":132,"threads_started":1}
W20260619 16:49:43.009884 9289 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:43.003447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39965 (local address 127.0.98.62:43111)
0619 16:49:43.003503 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:43.003506 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:43.003682 (+ 176us) server_negotiation.cc:407] Connection header received
0619 16:49:43.003770 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:43.003770 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:43.003785 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:43.004816 (+ 1031us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:43.005189 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.005723 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.006557 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.006753 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.006820 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:43.009816 (+ 2996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39965: BlockingRecv error: recv got EOF from 127.0.98.1:39965 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Jun 19 16:49:43 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:43 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:43.014919 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:43.010409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.1:46735)
0619 16:49:43.010504 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:43.010515 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:43.010570 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:43.010702 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:43.010703 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:43.010854 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:43.010904 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:43.010906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.011560 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.011561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:43.012092 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:43.012094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.012425 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.012426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:43.012488 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:43.012940 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:43.012944 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:43.014825 (+ 1881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":73}
W20260619 16:49:43.015012 9289 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:43.010461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46735 (local address 127.0.98.62:43111)
0619 16:49:43.010513 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:43.010515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:43.010551 (+ 36us) server_negotiation.cc:407] Connection header received
0619 16:49:43.010634 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:43.010635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:43.010646 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:43.010693 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:43.010972 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.011486 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.012167 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.012363 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.012422 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:43.014941 (+ 2519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46735: BlockingRecv error: recv got EOF from 127.0.98.1:46735 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260619 16:49:43.015372 9286 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 16:49:43.100044 9290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:43.100307 9290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:43.100335 9290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:43.101879 9290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:43.101965 9290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:43.101998 9290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:49:43.104274 9290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41797
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45159
--tserver_master_addrs=127.0.98.62:43111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:43.104593 9290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:43.104859 9290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:43.111287 9299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:43.111296 9297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:43.111294 9296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:43.111356 9290 server_base.cc:1034] running on GCE node
I20260619 16:49:43.111877 9290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:43.112161 9290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:43.113343 9290 hybrid_clock.cc:648] HybridClock initialized: now 1781887783113327 us; error 34 us; skew 500 ppm
Jun 19 16:49:43 dist-test-slave-jsw9 krb5kdc[8555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887783, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:43.116668 9290 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 16:49:43.117396 9290 webserver.cc:460] Webserver started at http://127.0.98.2:45159/ using document root <none> and password file <none>
I20260619 16:49:43.117660 9290 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:43.117746 9290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:43.119529 9290 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:43.120260 9306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:43.120536 9290 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:43.120639 9290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-1/wal
uuid: "28a05403b261428ba7052d0df43d97c0"
format_stamp: "Formatted at 2026-06-19 16:49:42 on dist-test-slave-jsw9"
server_key: "4c58b86f55df333d01adbdecdcebe1e9"
server_key_iv: "70ca456daadc828e09bfd1185ae466df"
server_key_version: "encryptionkey@0"
I20260619 16:49:43.121105 9290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:43.137760 9290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:43.140727 9290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:43.140926 9290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:43.141197 9290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:43.141543 9290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:43.141587 9290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:43.141620 9290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:43.141644 9290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:43.148849 9290 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41797
I20260619 16:49:43.148902 9419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41797 every 8 connection(s)
I20260619 16:49:43.149324 9290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 16:49:43 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:43 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 16:49:43.158390 9289 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:43.150234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49123 (local address 127.0.98.62:43111)
0619 16:49:43.150289 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:43.150291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:43.150548 (+ 257us) server_negotiation.cc:407] Connection header received
0619 16:49:43.151206 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:43.151208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:43.151269 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:43.151325 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:43.152594 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.153152 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.154417 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.154620 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.154684 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:43.158287 (+ 3603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49123: BlockingRecv error: recv got EOF from 127.0.98.2:49123 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 16:49:43.158424 9422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:43.150215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.2:49123)
0619 16:49:43.150403 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:43.150448 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:43.151079 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:43.151427 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:43.151431 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:43.152101 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:43.152451 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:43.152459 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.153256 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.153257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:43.154302 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:43.154305 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.154699 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.154700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:43.154860 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:43.155499 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:43.155510 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:43.158120 (+ 2610us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":93,"threads_started":1}
W20260619 16:49:43.158726 9420 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43111 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:43111: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260619 16:49:43.159368 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 9290
I20260619 16:49:43.159826 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33563
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43111
--webserver_port=32777
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:49:43 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:43 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:43.164640 9422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:43.159155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.2:57529)
0619 16:49:43.159220 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:43.159235 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:43.159304 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:43.159455 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:43.159457 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:43.159625 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:43.159768 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:43.159773 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.160631 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.160632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:43.161405 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:43.161409 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.161926 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.161927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:43.162082 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:43.162627 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:43.162632 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:43.164574 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 16:49:43.164798 9289 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:43.159178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57529 (local address 127.0.98.62:43111)
0619 16:49:43.159239 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:43.159241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:43.159284 (+ 43us) server_negotiation.cc:407] Connection header received
0619 16:49:43.159368 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:43.159369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:43.159378 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:43.159440 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:43.159956 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.160526 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.161523 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.161829 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.161912 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:43.164721 (+ 2809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57529: BlockingRecv error: recv got EOF from 127.0.98.2:57529 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Jun 19 16:49:43 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:43 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:43.170550 9422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:43.165156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.2:43883)
0619 16:49:43.165272 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:43.165338 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:43.165398 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:43.165583 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:43.165584 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:43.165820 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:43.165887 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:43.165891 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.166900 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.166901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:43.167471 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:43.167473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.167931 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.167932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:43.167992 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:43.168620 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:43.168625 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:43.170480 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":125,"threads_started":1}
W20260619 16:49:43.170698 9289 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:43.165394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43883 (local address 127.0.98.62:43111)
0619 16:49:43.165436 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:43.165438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:43.165449 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:49:43.165480 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:43.165481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:43.165493 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:43.165570 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:43.165980 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.166798 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.167553 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.167837 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.167915 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:43.170621 (+ 2706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43883: BlockingRecv error: recv got EOF from 127.0.98.2:43883 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260619 16:49:43.170897 9420 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 16:49:43.254578 9423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:43.254925 9423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:43.254969 9423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:43.256464 9423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:43.256520 9423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:43.256565 9423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:49:43.258297 9423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33563
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=32777
--tserver_master_addrs=127.0.98.62:43111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:43.258599 9423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:43.258858 9423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:43.265017 9429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:43.265017 9432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:43.265017 9430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:43.265331 9423 server_base.cc:1034] running on GCE node
I20260619 16:49:43.265612 9423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:43.265882 9423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:43.267062 9423 hybrid_clock.cc:648] HybridClock initialized: now 1781887783267041 us; error 36 us; skew 500 ppm
Jun 19 16:49:43 dist-test-slave-jsw9 krb5kdc[8555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887783, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:49:43.269905 9423 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 16:49:43.270478 9423 webserver.cc:460] Webserver started at http://127.0.98.3:32777/ using document root <none> and password file <none>
I20260619 16:49:43.270730 9423 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:43.270798 9423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:43.272397 9423 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 16:49:43.273203 9439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:43.273408 9423 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:43.273514 9423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-2/wal
uuid: "99a6b200fc8b4cac919fff622a4941c9"
format_stamp: "Formatted at 2026-06-19 16:49:42 on dist-test-slave-jsw9"
server_key: "821ceedb2ef805f3c53ab434b60338b0"
server_key_iv: "bb0e3429ce01525f4107000092a78c37"
server_key_version: "encryptionkey@0"
I20260619 16:49:43.273976 9423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:43.293493 9423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:43.296204 9423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:43.296380 9423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:43.296665 9423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:43.297155 9423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:43.297278 9423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:43.297320 9423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:43.297338 9423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:43.305019 9423 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33563
I20260619 16:49:43.305109 9552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33563 every 8 connection(s)
I20260619 16:49:43.305511 9423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:49:43.306915 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 9423
Jun 19 16:49:43 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:43 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:43.314728 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:43.306493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.3:59323)
0619 16:49:43.306727 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:43.306788 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:43.307503 (+ 715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:43.307832 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:43.307837 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:43.308686 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:43.308930 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:43.308942 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.309708 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.309709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:43.310995 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:43.310999 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.311423 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.311424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:43.311596 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:43.312055 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:43.312069 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:43.314522 (+ 2453us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":102,"threads_started":1}
W20260619 16:49:43.314718 9289 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:43.306466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59323 (local address 127.0.98.62:43111)
0619 16:49:43.306549 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:43.306552 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:43.306891 (+ 339us) server_negotiation.cc:407] Connection header received
0619 16:49:43.307649 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:43.307651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:43.307664 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:43.307744 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:43.309034 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.309609 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.311074 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.311337 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.311390 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:43.314645 (+ 3255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59323: BlockingRecv error: recv got EOF from 127.0.98.3:59323 (error 108)
Metrics: {"mutex_wait_us":14,"server-negotiator.queue_time_us":21}
W20260619 16:49:43.315268 9553 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43111 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:43111: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260619 16:49:43.316006 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:43.307476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51662 (local address 127.0.98.62:43111)
0619 16:49:43.307582 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:43.307584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:43.307593 (+ 9us) server_negotiation.cc:407] Connection header received
0619 16:49:43.307654 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:43.307655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:43.307664 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:43.307746 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:43.308579 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.309251 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.310341 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.310560 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.310617 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:43.312313 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:43.312319 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:43.312319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:49:43.312331 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:43.314095 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:43.314555 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:43.314559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:43.314560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:43.314717 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:49:43.315067 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:49:43.315068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:49:43.315069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:49:43.315345 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:43.315508 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:43.315772 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 16:49:43.315909 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":87,"thread_start_us":53,"threads_started":1}
Jun 19 16:49:43 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:43 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:43.320673 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:43.315678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.3:42429)
0619 16:49:43.315733 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:43.315744 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:43.315782 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:43.315896 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:43.315897 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:43.316057 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:43.316131 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:43.316133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.316883 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.316884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:43.317472 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:43.317475 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.317901 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.317902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:43.318014 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:43.318673 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:43.318677 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:43.320607 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260619 16:49:43.320816 9289 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:43.315721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42429 (local address 127.0.98.62:43111)
0619 16:49:43.315757 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:43.315759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:43.315768 (+ 9us) server_negotiation.cc:407] Connection header received
0619 16:49:43.315827 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:43.315828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:43.315840 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:43.315886 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:43.316205 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.316823 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.317569 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.317800 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.317867 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:43.320724 (+ 2857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42429: BlockingRecv error: recv got EOF from 127.0.98.3:42429 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Jun 19 16:49:43 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:43 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:43.325649 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:43.321020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.3:50859)
0619 16:49:43.321066 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:43.321077 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:43.321116 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:43.321271 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:43.321272 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:43.321431 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:43.321474 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:43.321475 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.322127 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.322128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:43.322683 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:43.322685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.323019 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.323021 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:43.323069 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:43.323551 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:43.323556 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:43.325580 (+ 2024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 16:49:43.325788 9289 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:43.321114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50859 (local address 127.0.98.62:43111)
0619 16:49:43.321173 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:43.321174 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:43.321182 (+ 8us) server_negotiation.cc:407] Connection header received
0619 16:49:43.321208 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:43.321208 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:43.321217 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:43.321262 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:43.321544 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.322058 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.322755 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:43.322957 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:43.323019 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:43.325717 (+ 2698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50859: BlockingRecv error: recv got EOF from 127.0.98.3:50859 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260619 16:49:43.325866 9553 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 16:49:44 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:44 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:44.020915 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:44.015869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.1:54481)
0619 16:49:44.016051 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:44.016065 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:44.016164 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:44.016362 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:44.016363 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:44.016554 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:44.016639 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:44.016659 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:44.017423 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:44.017424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:44.018022 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:44.018024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:44.018384 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:44.018384 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:44.018443 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:44.018992 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:44.018996 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:44.020843 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":86,"threads_started":1}
W20260619 16:49:44.021072 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:44.015947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54481 (local address 127.0.98.62:43111)
0619 16:49:44.016055 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:44.016057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:44.016141 (+ 84us) server_negotiation.cc:407] Connection header received
0619 16:49:44.016232 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:44.016233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:44.016250 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:44.016317 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:44.016754 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:44.017323 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:44.018096 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:44.018314 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:44.018367 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:44.020978 (+ 2611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54481: BlockingRecv error: recv got EOF from 127.0.98.1:54481 (error 108)
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":55,"threads_started":1}
Jun 19 16:49:44 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:44 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:44.177516 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:44.171541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.2:46659)
0619 16:49:44.171750 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:44.171764 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:44.171852 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:44.172003 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:44.172004 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:44.172200 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:44.172276 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:44.172278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:44.173229 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:44.173230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:44.173865 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:44.173868 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:44.174306 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:44.174307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:44.174370 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:44.175043 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:44.175049 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:44.177400 (+ 2351us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":93,"threads_started":1}
W20260619 16:49:44.177556 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:44.171692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46659 (local address 127.0.98.62:43111)
0619 16:49:44.171765 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:44.171767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:44.171838 (+ 71us) server_negotiation.cc:407] Connection header received
0619 16:49:44.171920 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:44.171921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:44.171934 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:44.171987 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:44.172380 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:44.173127 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:44.173961 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:44.174231 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:44.174300 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:44.177499 (+ 3199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46659: BlockingRecv error: recv got EOF from 127.0.98.2:46659 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 19 16:49:44 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:44 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:44.331637 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:44.326490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.3:43897)
0619 16:49:44.326639 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:44.326652 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:44.326732 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:44.326891 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:44.326892 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:44.327087 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:44.327153 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:44.327155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:44.327920 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:44.327921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:44.328522 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:44.328524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:44.328923 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:44.328924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:44.328971 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:44.329552 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:44.329557 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:44.331549 (+ 1992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
W20260619 16:49:44.331772 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:44.326582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43897 (local address 127.0.98.62:43111)
0619 16:49:44.326673 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:44.326675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:44.326709 (+ 34us) server_negotiation.cc:407] Connection header received
0619 16:49:44.326783 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:44.326784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:44.326797 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:44.326864 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:44.327246 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:44.327810 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:44.328619 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:44.328823 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:44.328923 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:44.331689 (+ 2766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43897: BlockingRecv error: recv got EOF from 127.0.98.3:43897 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 16:49:45 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:45 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:45.027575 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:45.021625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.1:46575)
0619 16:49:45.021829 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:45.021842 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:45.021918 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:45.022174 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:45.022176 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:45.022404 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:45.022484 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:45.022487 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:45.023281 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:45.023282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:45.023916 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:45.023920 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:45.024541 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:45.024542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:45.024620 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:45.025292 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:45.025298 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:45.027498 (+ 2200us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":90,"threads_started":1}
W20260619 16:49:45.027783 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:45.021763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46575 (local address 127.0.98.62:43111)
0619 16:49:45.021897 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:45.021899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:45.021916 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:49:45.021978 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:45.021979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:45.021990 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:45.022058 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:45.022596 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:45.023151 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:45.024064 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:45.024437 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:45.024509 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:45.027660 (+ 3151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46575: BlockingRecv error: recv got EOF from 127.0.98.1:46575 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":66,"threads_started":1}
Jun 19 16:49:45 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:45 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:45.184752 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:45.178124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.2:51133)
0619 16:49:45.178396 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:45.178408 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:45.178595 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:45.178746 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:45.178747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:45.179028 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:45.179135 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:45.179139 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:45.179883 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:45.179884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:45.180694 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:45.180704 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:45.181190 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:45.181191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:45.181252 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:45.181846 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:45.181852 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:45.184674 (+ 2822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":142,"threads_started":1}
W20260619 16:49:45.184907 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:45.178242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51133 (local address 127.0.98.62:43111)
0619 16:49:45.178307 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:45.178310 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:45.178569 (+ 259us) server_negotiation.cc:407] Connection header received
0619 16:49:45.178665 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:45.178666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:45.178682 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:45.178733 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:45.179215 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:45.179796 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:45.180885 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:45.181112 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:45.181178 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:45.184836 (+ 3658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51133: BlockingRecv error: recv got EOF from 127.0.98.2:51133 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Jun 19 16:49:45 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:45 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:45.338045 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:45.332348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.3:37223)
0619 16:49:45.332526 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:45.332547 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:45.332637 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:45.332786 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:45.332788 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:45.333004 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:45.333070 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:45.333073 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:45.333799 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:45.333800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:45.334381 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:45.334384 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:45.334735 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:45.334736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:45.334794 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:45.335430 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:45.335436 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:45.337966 (+ 2530us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":115,"threads_started":1}
W20260619 16:49:45.338172 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:45.332517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37223 (local address 127.0.98.62:43111)
0619 16:49:45.332613 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:45.332616 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:45.332626 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:49:45.332702 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:45.332704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:45.332720 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:45.332774 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:45.333158 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:45.333711 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:45.334466 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:45.334672 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:45.334731 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:45.338099 (+ 3368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37223: BlockingRecv error: recv got EOF from 127.0.98.3:37223 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 19 16:49:46 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:46 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:46.038604 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:46.028092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.1:39513)
0619 16:49:46.028307 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:46.028324 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:46.028394 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:46.028654 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:46.028656 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:46.028881 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:46.028951 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:46.028954 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:46.029799 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:46.029801 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:46.030445 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:46.030448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:46.031004 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:46.031005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:46.031079 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:46.035950 (+ 4871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:46.035958 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:46.038523 (+ 2565us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":75,"threads_started":1}
W20260619 16:49:46.038854 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:46.028244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39513 (local address 127.0.98.62:43111)
0619 16:49:46.028409 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:46.028412 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:46.028422 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:49:46.028455 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:46.028455 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:46.028470 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:46.028574 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:46.029072 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:46.029663 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:46.030566 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:46.030845 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:46.030923 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:46.038760 (+ 7837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39513: BlockingRecv error: recv got EOF from 127.0.98.1:39513 (error 108)
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":68,"threads_started":1}
Jun 19 16:49:46 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:46 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:46.191040 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:46.185460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.2:43419)
0619 16:49:46.185619 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:46.185635 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:46.185729 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:46.185905 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:46.185906 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:46.186112 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:46.186176 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:46.186178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:46.186950 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:46.186952 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:46.187650 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:46.187653 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:46.188091 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:46.188092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:46.188150 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:46.188682 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:46.188687 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:46.190950 (+ 2263us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":85,"threads_started":1}
W20260619 16:49:46.191164 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:46.185605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43419 (local address 127.0.98.62:43111)
0619 16:49:46.185699 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:46.185701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:46.185714 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:49:46.185820 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:46.185821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:46.185837 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:46.185885 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:46.186288 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:46.186875 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:46.187745 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:46.187984 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:46.188051 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:46.191095 (+ 3044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43419: BlockingRecv error: recv got EOF from 127.0.98.2:43419 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 16:49:46 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:46 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:46.344475 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:46.338812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.3:58115)
0619 16:49:46.339021 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:46.339034 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:46.339116 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:46.339287 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:46.339288 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:46.339524 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:46.339588 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:46.339591 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:46.340313 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:46.340315 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:46.340875 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:46.340876 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:46.341213 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:46.341214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:46.341273 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:46.341797 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:46.341809 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:46.344231 (+ 2422us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":82,"threads_started":1}
W20260619 16:49:46.344568 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:46.338887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58115 (local address 127.0.98.62:43111)
0619 16:49:46.338938 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:46.338941 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:46.339103 (+ 162us) server_negotiation.cc:407] Connection header received
0619 16:49:46.339204 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:46.339206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:46.339222 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:46.339272 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:46.339662 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:46.340234 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:46.340957 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:46.341150 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:46.341207 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:46.344501 (+ 3294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58115: BlockingRecv error: recv got EOF from 127.0.98.3:58115 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Jun 19 16:49:47 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:47 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:47.045411 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:47.039339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.1:46729)
0619 16:49:47.039644 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:47.039658 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:47.039777 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:47.039949 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:47.039950 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:47.040323 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:47.040396 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:47.040401 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:47.041258 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:47.041259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:47.041932 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:47.041936 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:47.042447 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:47.042448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:47.042513 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:47.043287 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:47.043293 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:47.045314 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":83,"threads_started":1}
W20260619 16:49:47.045580 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:47.039454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46729 (local address 127.0.98.62:43111)
0619 16:49:47.039641 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:47.039644 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:47.039764 (+ 120us) server_negotiation.cc:407] Connection header received
0619 16:49:47.039842 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:47.039843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:47.039859 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:47.039930 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:47.040542 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:47.041091 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:47.042089 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:47.042364 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:47.042440 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:47.045479 (+ 3039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46729: BlockingRecv error: recv got EOF from 127.0.98.1:46729 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":76,"threads_started":1}
Jun 19 16:49:47 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:47 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:47.196511 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:47.191631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.2:60569)
0619 16:49:47.191809 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:47.191821 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:47.191908 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:47.192071 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:47.192072 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:47.192272 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:47.192335 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:47.192337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:47.193052 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:47.193053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:47.193611 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:47.193613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:47.193955 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:47.193956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:47.194007 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:47.194557 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:47.194562 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:47.196437 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":89,"threads_started":1}
W20260619 16:49:47.196622 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:47.191685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60569 (local address 127.0.98.62:43111)
0619 16:49:47.191786 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:47.191805 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:47.191898 (+ 93us) server_negotiation.cc:407] Connection header received
0619 16:49:47.191987 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:47.191988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:47.192004 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:47.192058 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:47.192414 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:47.192971 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:47.193697 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:47.193889 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:47.193946 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:47.196555 (+ 2609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60569: BlockingRecv error: recv got EOF from 127.0.98.2:60569 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 16:49:47 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:47 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:47.351167 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:47.345146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.3:52463)
0619 16:49:47.345342 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:47.345357 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:47.345461 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:47.345730 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:47.345732 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:47.345966 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:47.346101 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:47.346106 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:47.347095 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:47.347097 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:47.347676 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:47.347678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:47.348025 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:47.348026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:47.348085 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:47.348646 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:47.348651 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:47.351090 (+ 2439us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":89,"threads_started":1}
W20260619 16:49:47.351342 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:47.345311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52463 (local address 127.0.98.62:43111)
0619 16:49:47.345428 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:47.345431 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:47.345443 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:49:47.345523 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:47.345524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:47.345539 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:47.345699 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:47.346412 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:47.347001 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:47.347764 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:47.347961 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:47.348020 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:47.351253 (+ 3233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52463: BlockingRecv error: recv got EOF from 127.0.98.3:52463 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Jun 19 16:49:48 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:48 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:48.052268 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:48.046150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.1:54333)
0619 16:49:48.046394 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:48.046418 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:48.046505 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:48.046740 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:48.046741 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:48.047001 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:48.047075 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:48.047077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:48.048081 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:48.048082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:48.048725 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:48.048728 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:48.049333 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:48.049334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:48.049413 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:48.050110 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:48.050117 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:48.052156 (+ 2039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":112,"threads_started":1}
W20260619 16:49:48.052363 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:48.046400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54333 (local address 127.0.98.62:43111)
0619 16:49:48.046546 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:48.046548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:48.046558 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:49:48.046590 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:48.046591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:48.046603 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:48.046675 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:48.047249 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:48.047970 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:48.048894 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:48.049146 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:48.049283 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:48.052290 (+ 3007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54333: BlockingRecv error: recv got EOF from 127.0.98.1:54333 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":75,"threads_started":1}
Jun 19 16:49:48 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:48 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:48.202219 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:48.196988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.2:55213)
0619 16:49:48.197157 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:48.197180 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:48.197265 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:48.197400 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:48.197401 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:48.197609 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:48.197694 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:48.197697 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:48.198438 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:48.198439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:48.199051 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:48.199054 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:48.199400 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:48.199401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:48.199453 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:48.200016 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:48.200022 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:48.202085 (+ 2063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":77,"threads_started":1}
W20260619 16:49:48.202354 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:48.197123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55213 (local address 127.0.98.62:43111)
0619 16:49:48.197185 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:48.197188 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:48.197256 (+ 68us) server_negotiation.cc:407] Connection header received
0619 16:49:48.197319 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:48.197320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:48.197335 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:48.197413 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:48.197774 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:48.198356 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:48.199132 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:48.199336 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:48.199400 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:48.202277 (+ 2877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55213: BlockingRecv error: recv got EOF from 127.0.98.2:55213 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 19 16:49:48 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:48 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:48.357479 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:48.351957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.3:57387)
0619 16:49:48.352144 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:48.352157 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:48.352237 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:48.352404 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:48.352405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:48.352665 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:48.352729 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:48.352732 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:48.353444 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:48.353445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:48.354075 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:48.354078 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:48.354607 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:48.354610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:48.354673 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:48.355243 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:48.355249 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:48.357405 (+ 2156us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":85,"threads_started":1}
W20260619 16:49:48.357628 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:48.352044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57387 (local address 127.0.98.62:43111)
0619 16:49:48.352122 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:48.352124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:48.352230 (+ 106us) server_negotiation.cc:407] Connection header received
0619 16:49:48.352305 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:48.352306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:48.352320 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:48.352363 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:48.352796 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:48.353346 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:48.354227 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:48.354469 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:48.354544 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:48.357547 (+ 3003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57387: BlockingRecv error: recv got EOF from 127.0.98.3:57387 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 16:49:49 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:49 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:49.058230 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:49.052954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.1:33479)
0619 16:49:49.053116 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:49.053128 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:49.053192 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:49.053355 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:49.053356 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:49.053543 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:49.053604 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:49.053606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.054350 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.054351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:49.054987 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:49.054989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.055434 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.055435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:49.055501 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:49.056081 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:49.056086 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:49.058110 (+ 2024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":88,"threads_started":1}
W20260619 16:49:49.058379 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:49.053059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33479 (local address 127.0.98.62:43111)
0619 16:49:49.053209 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:49.053211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:49.053221 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:49:49.053250 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:49.053250 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:49.053262 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:49.053337 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:49.053661 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.054256 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.055085 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.055296 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.055388 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:49.058297 (+ 2909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33479: BlockingRecv error: recv got EOF from 127.0.98.1:33479 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":70,"threads_started":1}
Jun 19 16:49:49 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:49 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:49.208747 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:49.202845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.2:51209)
0619 16:49:49.203147 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:49.203162 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:49.203246 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:49.203481 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:49.203483 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:49.203802 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:49.203871 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:49.203874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.204734 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.204736 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:49.205672 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:49.205677 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.206087 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.206088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:49.206136 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:49.206729 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:49.206734 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:49.208671 (+ 1937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":234,"threads_started":1}
W20260619 16:49:49.208875 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:49.203015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51209 (local address 127.0.98.62:43111)
0619 16:49:49.203113 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:49.203116 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:49.203271 (+ 155us) server_negotiation.cc:407] Connection header received
0619 16:49:49.203307 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:49.203338 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:49.203352 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:49.203407 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:49.203978 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.204574 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.205791 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.206011 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.206070 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:49.208805 (+ 2735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51209: BlockingRecv error: recv got EOF from 127.0.98.2:51209 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 16:49:49 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:49 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:49.364512 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:49.358234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.3:45207)
0619 16:49:49.358407 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:49.358422 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:49.358516 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:49.358690 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:49.358691 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:49.358889 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:49.358942 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:49.358945 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.359913 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.359914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:49.360469 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:49.360471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.360846 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.360847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:49.360899 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:49.361439 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:49.361444 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:49.364411 (+ 2967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":76,"threads_started":1}
W20260619 16:49:49.364840 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:49.358447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45207 (local address 127.0.98.62:43111)
0619 16:49:49.358529 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:49.358532 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:49.358543 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:49:49.358607 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:49.358608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:49.358622 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:49.358676 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:49.359256 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.359816 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.360584 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:49.360784 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:49.360843 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:49.364757 (+ 3914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45207: BlockingRecv error: recv got EOF from 127.0.98.3:45207 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 16:49:50 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:50 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:50.066408 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:50.058983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.1:60895)
0619 16:49:50.059196 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:50.059208 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:50.059288 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:50.059493 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:50.059495 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:50.059700 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:50.059759 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:50.059761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.060773 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.060774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:50.061458 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:50.061460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.061858 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.061859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:50.061932 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:50.062755 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:50.062760 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:50.066311 (+ 3551us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":102,"threads_started":1}
W20260619 16:49:50.066663 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:50.059181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60895 (local address 127.0.98.62:43111)
0619 16:49:50.059313 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:50.059316 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:50.059327 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:49:50.059359 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:50.059360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:50.059373 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:50.059439 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:50.059887 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.060580 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.061559 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.061770 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.061832 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:50.066581 (+ 4749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60895: BlockingRecv error: recv got EOF from 127.0.98.1:60895 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":60,"threads_started":1}
Jun 19 16:49:50 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:50 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:50.214387 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:50.209303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.2:46419)
0619 16:49:50.209477 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:50.209489 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:50.209564 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:50.209685 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:50.209686 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:50.209885 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:50.209947 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:50.209949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.210676 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.210678 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:50.211254 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:50.211257 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.211699 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.211700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:50.211755 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:50.212343 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:50.212348 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:50.214313 (+ 1965us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":81,"threads_started":1}
W20260619 16:49:50.214545 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:50.209430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46419 (local address 127.0.98.62:43111)
0619 16:49:50.209480 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:50.209482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:50.209553 (+ 71us) server_negotiation.cc:407] Connection header received
0619 16:49:50.209613 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:50.209614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:50.209626 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:50.209672 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:50.210009 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.210597 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.211361 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.211596 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.211656 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:50.214473 (+ 2817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46419: BlockingRecv error: recv got EOF from 127.0.98.2:46419 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Jun 19 16:49:50 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:50 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:50.371244 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:50.365329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.3:45707)
0619 16:49:50.365528 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:50.365542 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:50.365666 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:50.365825 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:50.365826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:50.366047 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:50.366131 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:50.366134 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.367170 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.367171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:50.367844 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:50.367847 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.368341 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.368342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:50.368419 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:50.368992 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:50.368998 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:50.371146 (+ 2148us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":97,"threads_started":1}
W20260619 16:49:50.371392 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:50.365496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45707 (local address 127.0.98.62:43111)
0619 16:49:50.365563 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:50.365565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:50.365638 (+ 73us) server_negotiation.cc:407] Connection header received
0619 16:49:50.365732 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:50.365733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:50.365749 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:50.365812 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:50.366246 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.367004 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.367972 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:50.368213 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:50.368282 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:50.371310 (+ 3028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45707: BlockingRecv error: recv got EOF from 127.0.98.3:45707 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 19 16:49:51 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:51 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:51.076424 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:51.067076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.1:50605)
0619 16:49:51.067271 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:51.067283 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:51.067356 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:51.067504 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:51.067505 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:51.067693 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:51.067755 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:51.067757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.068594 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.068595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:51.069167 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:51.069170 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.069623 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.069624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:51.069692 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:51.074000 (+ 4308us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:51.074008 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:51.076345 (+ 2337us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":79,"threads_started":1}
W20260619 16:49:51.076642 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:51.067243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50605 (local address 127.0.98.62:43111)
0619 16:49:51.067355 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:51.067357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:51.067389 (+ 32us) server_negotiation.cc:407] Connection header received
0619 16:49:51.067419 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:51.067421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:51.067432 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:51.067487 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:51.067873 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.068499 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.069246 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.069534 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.069604 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:51.076566 (+ 6962us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50605: BlockingRecv error: recv got EOF from 127.0.98.1:50605 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":49,"threads_started":1}
Jun 19 16:49:51 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:51 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:51.220407 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:51.214960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.2:36573)
0619 16:49:51.215109 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:51.215122 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:51.215189 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:51.215309 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:51.215309 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:51.215502 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:51.215597 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:51.215599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.216531 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.216532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:51.217099 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:51.217101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.217517 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.217518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:51.217568 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:51.218088 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:51.218093 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:51.220330 (+ 2237us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":109,"threads_started":1}
W20260619 16:49:51.220655 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:51.215067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36573 (local address 127.0.98.62:43111)
0619 16:49:51.215126 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:51.215129 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:51.215172 (+ 43us) server_negotiation.cc:407] Connection header received
0619 16:49:51.215239 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:51.215239 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:51.215251 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:51.215303 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:51.215683 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.216407 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.217221 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.217427 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.217513 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:51.220572 (+ 3059us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36573: BlockingRecv error: recv got EOF from 127.0.98.2:36573 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jun 19 16:49:51 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:51 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:51.377162 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:51.371962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.3:42275)
0619 16:49:51.372138 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:51.372152 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:51.372246 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:51.372409 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:51.372410 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:51.372680 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:51.372749 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:51.372752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.373569 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.373571 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:51.374127 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:51.374129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.374542 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.374543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:51.374610 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:51.375131 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:51.375136 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:51.377066 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":103,"threads_started":1}
W20260619 16:49:51.377269 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:51.372079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42275 (local address 127.0.98.62:43111)
0619 16:49:51.372142 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:51.372144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:51.372236 (+ 92us) server_negotiation.cc:407] Connection header received
0619 16:49:51.372317 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:51.372318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:51.372334 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:51.372393 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:51.372858 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.373422 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.374244 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:51.374475 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:51.374541 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:51.377204 (+ 2663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42275: BlockingRecv error: recv got EOF from 127.0.98.3:42275 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Jun 19 16:49:52 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:52 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:52.083256 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:52.077091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.1:35965)
0619 16:49:52.077302 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:52.077328 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:52.077452 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:52.077707 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:52.077708 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:52.077906 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:52.077967 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:52.077969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:52.078826 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:52.078828 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:52.079432 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:52.079435 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:52.080011 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:52.080012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:52.080093 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:52.080722 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:52.080728 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:52.083166 (+ 2438us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":104,"threads_started":1}
W20260619 16:49:52.083364 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:52.077293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35965 (local address 127.0.98.62:43111)
0619 16:49:52.077416 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:52.077419 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:52.077433 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:49:52.077540 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:52.077542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:52.077568 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:52.077651 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:52.078095 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:52.078739 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:52.079537 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:52.079800 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:52.079865 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:52.083298 (+ 3433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35965: BlockingRecv error: recv got EOF from 127.0.98.1:35965 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":66,"threads_started":1}
Jun 19 16:49:52 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:52 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:52.226639 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:52.221132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.2:39679)
0619 16:49:52.221300 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:52.221313 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:52.221390 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:52.221571 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:52.221572 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:52.221765 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:52.221829 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:52.221830 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:52.222678 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:52.222679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:52.223245 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:52.223247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:52.223664 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:52.223665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:52.223707 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:52.224280 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:52.224285 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:52.226543 (+ 2258us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":99,"threads_started":1}
W20260619 16:49:52.226708 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:52.221224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39679 (local address 127.0.98.62:43111)
0619 16:49:52.221329 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:52.221331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:52.221371 (+ 40us) server_negotiation.cc:407] Connection header received
0619 16:49:52.221453 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:52.221455 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:52.221467 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:52.221551 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:52.221929 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:52.222583 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:52.223392 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:52.223599 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:52.223659 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:52.226647 (+ 2988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39679: BlockingRecv error: recv got EOF from 127.0.98.2:39679 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 19 16:49:52 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:52 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:52.383162 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:52.377791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.3:47961)
0619 16:49:52.377947 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:52.377963 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:52.378034 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:52.378277 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:52.378278 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:52.378513 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:52.378577 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:52.378579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:52.379346 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:52.379347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:52.379942 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:52.379944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:52.380258 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:52.380259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:52.380307 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:52.380886 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:52.380890 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:52.383068 (+ 2178us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":75,"threads_started":1}
W20260619 16:49:52.383330 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:52.377896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47961 (local address 127.0.98.62:43111)
0619 16:49:52.377959 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:52.377961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:52.378048 (+ 87us) server_negotiation.cc:407] Connection header received
0619 16:49:52.378087 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:52.378088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:52.378103 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:52.378231 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:52.378698 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:52.379260 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:52.380004 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:52.380194 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:52.380245 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:52.383190 (+ 2945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47961: BlockingRecv error: recv got EOF from 127.0.98.3:47961 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Jun 19 16:49:53 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:53 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:53.090752 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.083846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.1:54105)
0619 16:49:53.084034 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:53.084051 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:53.084133 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:53.084340 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:53.084341 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:53.084544 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:53.084628 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:53.084630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.086086 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.086087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:53.086951 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:53.086955 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.087580 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.087581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:53.087636 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:53.088285 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:53.088292 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:53.090651 (+ 2359us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":92,"threads_started":1}
W20260619 16:49:53.090849 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:53.083981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54105 (local address 127.0.98.62:43111)
0619 16:49:53.084162 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:53.084164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:53.084175 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:49:53.084206 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:53.084207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:53.084220 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:53.084303 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:53.084742 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.085924 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.087089 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.087470 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.087581 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:53.090773 (+ 3192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54105: BlockingRecv error: recv got EOF from 127.0.98.1:54105 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":85,"threads_started":1}
Jun 19 16:49:53 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:53 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:53.232772 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.227239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.2:42525)
0619 16:49:53.227468 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:53.227481 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:53.227561 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:53.227707 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:53.227707 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:53.227902 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:53.227966 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:53.227968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.228713 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.228714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:53.229290 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:53.229292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.229670 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.229671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:53.229718 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:53.230430 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:53.230435 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:53.232692 (+ 2257us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":140,"threads_started":1}
W20260619 16:49:53.232937 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:53.227360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42525 (local address 127.0.98.62:43111)
0619 16:49:53.227424 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:53.227426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:53.227579 (+ 153us) server_negotiation.cc:407] Connection header received
0619 16:49:53.227622 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:53.227623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:53.227637 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:53.227695 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:53.228073 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.228629 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.229383 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.229591 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.229656 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:53.232861 (+ 3205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42525: BlockingRecv error: recv got EOF from 127.0.98.2:42525 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 19 16:49:53 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:53 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:53.388948 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:53.383683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.3:41845)
0619 16:49:53.383843 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:53.383855 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:53.383912 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:53.384047 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:53.384048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:53.384515 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:53.384585 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:53.384588 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.385343 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.385344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:53.385941 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:53.385944 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.386358 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.386359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:53.386413 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:53.386946 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:53.386951 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:53.388868 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":81,"threads_started":1}
W20260619 16:49:53.389158 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:53.383818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41845 (local address 127.0.98.62:43111)
0619 16:49:53.383881 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:53.383883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:53.383896 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:49:53.383968 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:53.383969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:53.383984 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:53.384032 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:53.384695 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.385247 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.386031 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:53.386271 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:53.386331 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:53.389099 (+ 2768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41845: BlockingRecv error: recv got EOF from 127.0.98.3:41845 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 19 16:49:54 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:54 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:54.096717 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:54.091422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.1:34847)
0619 16:49:54.091578 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:54.091591 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:54.091654 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:54.091918 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:54.091919 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:54.092125 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:54.092187 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:54.092190 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:54.092966 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:54.092966 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:54.093551 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:54.093554 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:54.093905 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:54.093906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:54.093953 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:54.094518 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:54.094523 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:54.096638 (+ 2115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":89,"threads_started":1}
W20260619 16:49:54.096920 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:54.091557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34847 (local address 127.0.98.62:43111)
0619 16:49:54.091719 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:54.091721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:54.091730 (+ 9us) server_negotiation.cc:407] Connection header received
0619 16:49:54.091770 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:54.091771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:54.091783 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:54.091883 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:54.092286 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:54.092886 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:54.093629 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:54.093835 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:54.093892 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:54.096835 (+ 2943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34847: BlockingRecv error: recv got EOF from 127.0.98.1:34847 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":61,"threads_started":1}
Jun 19 16:49:54 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:54 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:54.240427 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:54.233531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.2:50731)
0619 16:49:54.233757 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:54.233775 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:54.233861 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:54.234002 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:54.234003 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:54.234302 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:54.234370 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:54.234373 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:54.235249 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:54.235251 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:54.235868 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:54.235870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:54.236528 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:54.236529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:54.236584 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:54.237264 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:54.237270 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:54.240346 (+ 3076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":100,"threads_started":1}
W20260619 16:49:54.240595 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:54.233707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50731 (local address 127.0.98.62:43111)
0619 16:49:54.233778 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:54.233780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:54.233847 (+ 67us) server_negotiation.cc:407] Connection header received
0619 16:49:54.233919 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:54.233920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:54.233933 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:54.233984 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:54.234428 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:54.235131 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:54.236019 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:54.236374 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:54.236529 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:54.240518 (+ 3989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50731: BlockingRecv error: recv got EOF from 127.0.98.2:50731 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 19 16:49:54 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:54 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:54.394980 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:54.389599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.3:51693)
0619 16:49:54.389834 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:54.389845 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:54.389917 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:54.390068 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:54.390068 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:54.390303 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:54.390363 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:54.390366 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:54.391079 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:54.391080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:54.391739 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:54.391742 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:54.392194 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:54.392195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:54.392264 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:54.392841 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:54.392846 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:54.394866 (+ 2020us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":136,"threads_started":1}
W20260619 16:49:54.395054 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:54.389719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51693 (local address 127.0.98.62:43111)
0619 16:49:54.389817 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:54.389819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:54.389911 (+ 92us) server_negotiation.cc:407] Connection header received
0619 16:49:54.389971 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:54.389972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:54.389987 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:54.390051 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:54.390429 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:54.390991 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:54.391838 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:54.392102 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:54.392168 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:54.394990 (+ 2822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51693: BlockingRecv error: recv got EOF from 127.0.98.3:51693 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 16:49:55 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:55 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:55.103196 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:55.097407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.1:42219)
0619 16:49:55.097583 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:55.097595 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:55.097656 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:55.097965 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:55.097967 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:55.098354 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:55.098423 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:55.098426 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:55.099284 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:55.099285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:55.099873 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:55.099875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:55.100272 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:55.100273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:55.100324 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:55.100918 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:55.100923 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:55.103121 (+ 2198us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":86,"threads_started":1}
W20260619 16:49:55.103343 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:55.097618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42219 (local address 127.0.98.62:43111)
0619 16:49:55.097763 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:55.097766 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:55.097776 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:49:55.097806 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:55.097807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:55.097819 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:55.097913 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:55.098601 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:55.099173 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:55.099963 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:55.100183 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:55.100241 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:55.103253 (+ 3012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42219: BlockingRecv error: recv got EOF from 127.0.98.1:42219 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":73,"threads_started":1}
Jun 19 16:49:55 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:55 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:55.247032 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:55.241216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.2:48285)
0619 16:49:55.241402 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:55.241416 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:55.241493 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:55.241654 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:55.241655 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:55.241859 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:55.241924 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:55.241926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:55.242832 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:55.242834 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:55.243407 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:55.243409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:55.243848 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:55.243849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:55.243901 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:55.244516 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:55.244524 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:55.246887 (+ 2363us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":92,"threads_started":1}
W20260619 16:49:55.247151 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:55.241345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48285 (local address 127.0.98.62:43111)
0619 16:49:55.241457 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:55.241459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:55.241478 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:49:55.241567 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:55.241568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:55.241582 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:55.241637 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:55.242013 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:55.242720 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:55.243516 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:55.243753 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:55.243830 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:55.247089 (+ 3259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48285: BlockingRecv error: recv got EOF from 127.0.98.2:48285 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 19 16:49:55 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:55 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:55.401307 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:55.395682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.3:41367)
0619 16:49:55.395873 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:55.395885 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:55.395956 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:55.396140 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:55.396141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:55.396404 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:55.396472 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:55.396475 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:55.397272 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:55.397273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:55.397980 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:55.397984 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:55.398472 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:55.398473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:55.398528 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:55.399101 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:55.399106 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:55.401219 (+ 2113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":113,"threads_started":1}
W20260619 16:49:55.401459 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:55.395811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41367 (local address 127.0.98.62:43111)
0619 16:49:55.395893 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:55.395895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:55.395954 (+ 59us) server_negotiation.cc:407] Connection header received
0619 16:49:55.396045 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:55.396046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:55.396063 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:55.396127 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:55.396605 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:55.397164 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:55.398079 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:55.398345 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:55.398407 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:55.401371 (+ 2964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41367: BlockingRecv error: recv got EOF from 127.0.98.3:41367 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 16:49:56 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:56 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:56.112864 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:56.103956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.1:41979)
0619 16:49:56.104153 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:56.104168 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:56.104258 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:56.104598 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:56.104599 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:56.104802 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:56.104870 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:56.104872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:56.105754 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:56.105754 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:56.106580 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:56.106583 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:56.106976 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:56.106977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:56.107031 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:56.110573 (+ 3542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:56.110581 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:56.112755 (+ 2174us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":86,"threads_started":1}
W20260619 16:49:56.112982 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:56.104123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41979 (local address 127.0.98.62:43111)
0619 16:49:56.104418 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:56.104420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:56.104430 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:49:56.104460 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:56.104461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:56.104473 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:56.104545 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:56.104972 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:56.105650 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:56.106655 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:56.106905 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:56.106967 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:56.112896 (+ 5929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41979: BlockingRecv error: recv got EOF from 127.0.98.1:41979 (error 108)
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":149,"threads_started":1}
Jun 19 16:49:56 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:56 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:56.253022 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:56.247730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.2:44007)
0619 16:49:56.247927 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:56.247943 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:56.248014 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:56.248217 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:56.248218 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:56.248414 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:56.248476 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:56.248478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:56.249243 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:56.249245 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:56.249853 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:56.249856 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:56.250297 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:56.250298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:56.250368 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:56.250968 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:56.250974 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:56.252943 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":82,"threads_started":1}
W20260619 16:49:56.253146 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:56.247954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44007 (local address 127.0.98.62:43111)
0619 16:49:56.248046 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:56.248048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:56.248057 (+ 9us) server_negotiation.cc:407] Connection header received
0619 16:49:56.248116 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:56.248117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:56.248129 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:56.248182 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:56.248551 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:56.249130 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:56.249959 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:56.250182 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:56.250263 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:56.253072 (+ 2809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44007: BlockingRecv error: recv got EOF from 127.0.98.2:44007 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 16:49:56 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:56 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:56.407791 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:56.402033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.3:52933)
0619 16:49:56.402267 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:56.402282 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:56.402373 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:56.402582 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:56.402583 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:56.402818 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:56.402886 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:56.402888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:56.403658 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:56.403660 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:56.404255 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:56.404257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:56.404625 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:56.404626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:56.404686 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:56.405347 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:56.405353 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:56.407697 (+ 2344us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":91,"threads_started":1}
W20260619 16:49:56.407881 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:56.402180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52933 (local address 127.0.98.62:43111)
0619 16:49:56.402275 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:56.402278 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:56.402360 (+ 82us) server_negotiation.cc:407] Connection header received
0619 16:49:56.402448 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:56.402449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:56.402506 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:56.402566 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:56.402965 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:56.403564 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:56.404342 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:56.404555 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:56.404622 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:56.407816 (+ 3194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52933: BlockingRecv error: recv got EOF from 127.0.98.3:52933 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 16:49:57 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:57 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:57.118788 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:57.113468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.1:38799)
0619 16:49:57.113639 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:57.113653 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:57.113779 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:57.113936 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:57.113937 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:57.114143 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:57.114251 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:57.114254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:57.115091 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:57.115093 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:57.115662 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:57.115664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:57.116017 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:57.116018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:57.116076 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:57.116721 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:57.116727 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:57.118693 (+ 1966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":83,"threads_started":1}
W20260619 16:49:57.118901 9611 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:57.113533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38799 (local address 127.0.98.62:43111)
0619 16:49:57.113697 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:57.113699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:57.113770 (+ 71us) server_negotiation.cc:407] Connection header received
0619 16:49:57.113845 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:57.113846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:57.113861 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:57.113922 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:57.114330 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:57.114979 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:57.115751 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:57.115954 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:57.116010 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:57.118827 (+ 2817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38799: BlockingRecv error: recv got EOF from 127.0.98.1:38799 (error 108)
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":92,"threads_started":1}
Jun 19 16:49:57 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:57 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:57.258963 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:57.253704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.2:50853)
0619 16:49:57.253892 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:57.253903 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:57.253978 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:57.254215 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:57.254216 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:57.254421 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:57.254487 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:57.254489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:57.255238 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:57.255239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:57.255812 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:57.255814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:57.256201 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:57.256201 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:57.256252 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:57.256825 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:57.256830 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:57.258869 (+ 2039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":90,"threads_started":1}
W20260619 16:49:57.259022 9611 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:57.253803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50853 (local address 127.0.98.62:43111)
0619 16:49:57.253876 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:57.253878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:57.254036 (+ 158us) server_negotiation.cc:407] Connection header received
0619 16:49:57.254075 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:57.254076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:57.254091 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:57.254144 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:57.254571 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:57.255129 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:57.255914 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:57.256120 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:57.256185 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:57.258961 (+ 2776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50853: BlockingRecv error: recv got EOF from 127.0.98.2:50853 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 19 16:49:57 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:57 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:57.413578 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:57.408350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.3:43461)
0619 16:49:57.408534 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:57.408548 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:57.408670 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:57.408819 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:57.408820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:57.409037 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:57.409099 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:57.409101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:57.409886 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:57.409888 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:57.410494 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:57.410496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:57.410845 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:57.410846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:57.410905 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:57.411430 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:57.411435 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:57.413498 (+ 2063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":101,"threads_started":1}
W20260619 16:49:57.413806 9611 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:57.408471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43461 (local address 127.0.98.62:43111)
0619 16:49:57.408538 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:57.408540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:57.408614 (+ 74us) server_negotiation.cc:407] Connection header received
0619 16:49:57.408746 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:57.408747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:57.408763 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:57.408807 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:57.409188 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:57.409811 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:57.410563 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:57.410784 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:57.410841 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:57.413682 (+ 2841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43461: BlockingRecv error: recv got EOF from 127.0.98.3:43461 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jun 19 16:49:58 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:58 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:58.125523 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:58.119612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.1:44019)
0619 16:49:58.119810 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:58.119822 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:58.119886 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:58.120056 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:58.120056 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:58.120249 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:58.120307 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:58.120310 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:58.121184 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:58.121185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:58.121822 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:58.121824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:58.122303 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:58.122304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:58.122352 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:58.122941 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:58.122945 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:58.125446 (+ 2501us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":92,"threads_started":1}
W20260619 16:49:58.125801 9615 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:58.119779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44019 (local address 127.0.98.62:43111)
0619 16:49:58.119908 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:58.119910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:58.119919 (+ 9us) server_negotiation.cc:407] Connection header received
0619 16:49:58.119950 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:58.119951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:58.119963 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:58.120044 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:58.120386 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:58.121022 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:58.121946 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:58.122175 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:58.122305 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:58.125738 (+ 3433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44019: BlockingRecv error: recv got EOF from 127.0.98.1:44019 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
Jun 19 16:49:58 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:49:58 dist-test-slave-jsw9 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:49:58.266393 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:58.259831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43111 (local address 127.0.98.2:46899)
0619 16:49:58.260018 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:58.260033 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:58.260114 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:58.260337 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:58.260338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:58.260568 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:58.260680 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:58.260683 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:58.261764 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:58.261765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:58.262543 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:58.262548 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:58.262960 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:58.262961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:58.263019 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:58.263730 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:49:58.263737 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:58.266295 (+ 2558us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43111: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":109,"threads_started":1}
W20260619 16:49:58.266607 9615 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:58.259978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46899 (local address 127.0.98.62:43111)
0619 16:49:58.260082 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:58.260084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:58.260099 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:49:58.260196 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:58.260198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:58.260211 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:58.260272 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:58.260926 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:58.261620 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:58.262653 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:58.262867 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:58.262930 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:58.266490 (+ 3560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46899: BlockingRecv error: recv got EOF from 127.0.98.2:46899 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 16:49:58.316883 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 9138
I20260619 16:49:58.322548 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 9290
I20260619 16:49:58.327350 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 9423
I20260619 16:49:58.332218 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 9068
2026-06-19T16:49:58Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17785 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T16:49:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:49:58Z Disabled control of system clock
I20260619 16:49:58.363193 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43095
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43095
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 16:49:58.459316 9620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:58.459676 9620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:58.459720 9620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:58.461287 9620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:58.461338 9620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:58.461349 9620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:58.461361 9620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:58.461372 9620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:49:58.462951 9620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43095
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:58.463236 9620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:58.463557 9620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:58.469805 9625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:58.469806 9626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:58.469806 9628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:58.470295 9620 server_base.cc:1034] running on GCE node
I20260619 16:49:58.470587 9620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:58.470937 9620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:58.472118 9620 hybrid_clock.cc:648] HybridClock initialized: now 1781887798472088 us; error 41 us; skew 500 ppm
I20260619 16:49:58.473783 9620 webserver.cc:460] Webserver started at http://127.0.98.62:34539/ using document root <none> and password file <none>
I20260619 16:49:58.474073 9620 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:58.474128 9620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:58.474280 9620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:58.475631 9620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "da756e5274db4620ba3d81de2cb0962e"
format_stamp: "Formatted at 2026-06-19 16:49:58 on dist-test-slave-jsw9"
server_key: "3406db2c0aa94edce55c595481aed92d"
server_key_iv: "502ef4ed2fbc982e25b8a8e4a475934a"
server_key_version: "encryptionkey@0"
I20260619 16:49:58.476044 9620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "da756e5274db4620ba3d81de2cb0962e"
format_stamp: "Formatted at 2026-06-19 16:49:58 on dist-test-slave-jsw9"
server_key: "3406db2c0aa94edce55c595481aed92d"
server_key_iv: "502ef4ed2fbc982e25b8a8e4a475934a"
server_key_version: "encryptionkey@0"
I20260619 16:49:58.477703 9620 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 16:49:58.478744 9634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:58.478967 9620 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 16:49:58.479075 9620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "da756e5274db4620ba3d81de2cb0962e"
format_stamp: "Formatted at 2026-06-19 16:49:58 on dist-test-slave-jsw9"
server_key: "3406db2c0aa94edce55c595481aed92d"
server_key_iv: "502ef4ed2fbc982e25b8a8e4a475934a"
server_key_version: "encryptionkey@0"
I20260619 16:49:58.479149 9620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:58.491305 9620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:58.491722 9620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:58.491873 9620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:58.496518 9620 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43095
I20260619 16:49:58.496590 9686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43095 every 8 connection(s)
I20260619 16:49:58.496994 9620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:58.497872 9687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:58.499512 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 9620
I20260619 16:49:58.499620 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:49:58.499843 392 external_mini_cluster.cc:1428] Setting key 1e2cf106208364f6cf76737eab84f307
I20260619 16:49:58.501586 9687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da756e5274db4620ba3d81de2cb0962e: Bootstrap starting.
I20260619 16:49:58.502436 9687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da756e5274db4620ba3d81de2cb0962e: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:58.502741 9687 log.cc:824] T 00000000000000000000000000000000 P da756e5274db4620ba3d81de2cb0962e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:58.503464 9687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da756e5274db4620ba3d81de2cb0962e: No bootstrap required, opened a new log
I20260619 16:49:58.505820 9687 raft_consensus.cc:348] T 00000000000000000000000000000000 P da756e5274db4620ba3d81de2cb0962e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da756e5274db4620ba3d81de2cb0962e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43095 } }
I20260619 16:49:58.505986 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:58.500478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40182 (local address 127.0.98.62:43095)
0619 16:49:58.500710 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:58.500720 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:58.500832 (+ 112us) server_negotiation.cc:407] Connection header received
0619 16:49:58.501326 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:58.501334 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:58.501406 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:58.501539 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:58.501804 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:58.502941 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:58.503806 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:58.504250 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:58.504366 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:58.505132 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:58.505136 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:58.505137 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:58.505340 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:58.505378 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:58.505402 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:58.505541 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:58.505633 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 16:49:58.505764 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":109,"threads_started":1}
I20260619 16:49:58.506012 9687 raft_consensus.cc:374] T 00000000000000000000000000000000 P da756e5274db4620ba3d81de2cb0962e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:58.506049 9687 raft_consensus.cc:729] T 00000000000000000000000000000000 P da756e5274db4620ba3d81de2cb0962e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da756e5274db4620ba3d81de2cb0962e, State: Initialized, Role: FOLLOWER
I20260619 16:49:58.506240 9687 consensus_queue.cc:260] T 00000000000000000000000000000000 P da756e5274db4620ba3d81de2cb0962e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da756e5274db4620ba3d81de2cb0962e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43095 } }
I20260619 16:49:58.506661 9687 raft_consensus.cc:388] T 00000000000000000000000000000000 P da756e5274db4620ba3d81de2cb0962e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:58.506740 9687 raft_consensus.cc:482] T 00000000000000000000000000000000 P da756e5274db4620ba3d81de2cb0962e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:58.506798 9687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da756e5274db4620ba3d81de2cb0962e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:58.507520 9687 raft_consensus.cc:504] T 00000000000000000000000000000000 P da756e5274db4620ba3d81de2cb0962e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da756e5274db4620ba3d81de2cb0962e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43095 } }
I20260619 16:49:58.507627 9687 leader_election.cc:302] T 00000000000000000000000000000000 P da756e5274db4620ba3d81de2cb0962e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da756e5274db4620ba3d81de2cb0962e; no voters:
I20260619 16:49:58.507844 9687 leader_election.cc:288] T 00000000000000000000000000000000 P da756e5274db4620ba3d81de2cb0962e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:49:58.507944 9692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da756e5274db4620ba3d81de2cb0962e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:58.508112 9692 raft_consensus.cc:686] T 00000000000000000000000000000000 P da756e5274db4620ba3d81de2cb0962e [term 1 LEADER]: Becoming Leader. State: Replica: da756e5274db4620ba3d81de2cb0962e, State: Running, Role: LEADER
I20260619 16:49:58.508160 9687 sys_catalog.cc:564] T 00000000000000000000000000000000 P da756e5274db4620ba3d81de2cb0962e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:58.508260 9692 consensus_queue.cc:237] T 00000000000000000000000000000000 P da756e5274db4620ba3d81de2cb0962e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da756e5274db4620ba3d81de2cb0962e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43095 } }
I20260619 16:49:58.508759 9693 sys_catalog.cc:455] T 00000000000000000000000000000000 P da756e5274db4620ba3d81de2cb0962e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da756e5274db4620ba3d81de2cb0962e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da756e5274db4620ba3d81de2cb0962e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43095 } } }
I20260619 16:49:58.508770 9694 sys_catalog.cc:455] T 00000000000000000000000000000000 P da756e5274db4620ba3d81de2cb0962e [sys.catalog]: SysCatalogTable state changed. Reason: New leader da756e5274db4620ba3d81de2cb0962e. Latest consensus state: current_term: 1 leader_uuid: "da756e5274db4620ba3d81de2cb0962e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da756e5274db4620ba3d81de2cb0962e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43095 } } }
I20260619 16:49:58.508874 9693 sys_catalog.cc:458] T 00000000000000000000000000000000 P da756e5274db4620ba3d81de2cb0962e [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:58.508880 9694 sys_catalog.cc:458] T 00000000000000000000000000000000 P da756e5274db4620ba3d81de2cb0962e [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:58.509127 9701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:58.509774 9701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:58.511626 9701 catalog_manager.cc:1347] Generated new cluster ID: 589d173eacb34d21a64cc3fa8adea7b0
I20260619 16:49:58.511686 9701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:49:58.524484 9701 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:49:58.525429 9701 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:58.531479 9701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da756e5274db4620ba3d81de2cb0962e: Generated new TSK 0
I20260619 16:49:58.531780 9701 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:49:58.540783 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43095
--builtin_ntp_servers=127.0.98.20:44027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 16:49:58.651481 9711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:58.652058 9711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:58.652143 9711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:58.654050 9711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:58.654139 9711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:58.654181 9711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:49:58.656057 9711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:58.656385 9711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:58.656782 9711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:58.662633 9719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:58.662655 9717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:58.662967 9711 server_base.cc:1034] running on GCE node
W20260619 16:49:58.662753 9716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:58.663327 9711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:58.664000 9711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:58.665213 9711 hybrid_clock.cc:648] HybridClock initialized: now 1781887798665191 us; error 40 us; skew 500 ppm
I20260619 16:49:58.667089 9711 webserver.cc:460] Webserver started at http://127.0.98.1:40277/ using document root <none> and password file <none>
I20260619 16:49:58.667579 9711 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:58.667658 9711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:58.667819 9711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:58.669442 9711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-0/data/instance:
uuid: "77da1b63a97e4b10a8644690a556611d"
format_stamp: "Formatted at 2026-06-19 16:49:58 on dist-test-slave-jsw9"
server_key: "de3fa999c343f5e88ea6afceee66b279"
server_key_iv: "165231ab0aa65e0195d85003ca8a1d09"
server_key_version: "encryptionkey@0"
I20260619 16:49:58.669888 9711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-0/wal/instance:
uuid: "77da1b63a97e4b10a8644690a556611d"
format_stamp: "Formatted at 2026-06-19 16:49:58 on dist-test-slave-jsw9"
server_key: "de3fa999c343f5e88ea6afceee66b279"
server_key_iv: "165231ab0aa65e0195d85003ca8a1d09"
server_key_version: "encryptionkey@0"
I20260619 16:49:58.671761 9711 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 16:49:58.672714 9725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:58.672894 9711 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:58.672986 9711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-0/wal
uuid: "77da1b63a97e4b10a8644690a556611d"
format_stamp: "Formatted at 2026-06-19 16:49:58 on dist-test-slave-jsw9"
server_key: "de3fa999c343f5e88ea6afceee66b279"
server_key_iv: "165231ab0aa65e0195d85003ca8a1d09"
server_key_version: "encryptionkey@0"
I20260619 16:49:58.673063 9711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:58.684649 9711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:58.684998 9711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:58.685112 9711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:58.685339 9711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:58.685705 9711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:58.685740 9711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:58.685766 9711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:58.685779 9711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:58.692490 9711 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40585
I20260619 16:49:58.692543 9838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40585 every 8 connection(s)
I20260619 16:49:58.692938 9711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:49:58.699038 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:58.693765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34941 (local address 127.0.98.62:43095)
0619 16:49:58.693831 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:58.693833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:58.694115 (+ 282us) server_negotiation.cc:407] Connection header received
0619 16:49:58.694763 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:58.694764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:58.694788 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:58.694843 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:58.695676 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:58.696257 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:58.697439 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:58.697647 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:58.697709 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:58.698699 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:58.698701 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:58.698702 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:58.698709 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:58.698726 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:58.698730 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:58.698812 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:58.698905 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 16:49:58.698938 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 16:49:58.699304 9841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:58.693741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43095 (local address 127.0.98.1:34941)
0619 16:49:58.693946 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:58.693985 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:58.694643 (+ 658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:58.694930 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:58.694934 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:58.694945 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:58.695495 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:58.695556 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:58.696356 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:58.696357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:58.697324 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:58.697327 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:58.697742 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:58.697743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:58.697892 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:58.698533 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:49:58.698534 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:58.698571 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:49:58.698601 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:58.698821 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:58.698827 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:58.698832 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:49:58.698898 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 16:49:58.699028 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":92,"threads_started":1}
I20260619 16:49:58.699837 9839 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43095
I20260619 16:49:58.699955 9839 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:58.699988 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 9711
I20260619 16:49:58.700078 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:49:58.700196 9839 heartbeater.cc:511] Master 127.0.98.62:43095 requested a full tablet report, sending...
I20260619 16:49:58.700290 392 external_mini_cluster.cc:1428] Setting key f41583b3e969dfc2a48c85e4c44c9853
I20260619 16:49:58.700829 9651 ts_manager.cc:194] Registered new tserver with Master: 77da1b63a97e4b10a8644690a556611d (127.0.98.1:40585)
I20260619 16:49:58.701485 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43095
--builtin_ntp_servers=127.0.98.20:44027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 16:49:58.701617 9651 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:34941
W20260619 16:49:58.797505 9842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:58.797778 9842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:58.797806 9842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:58.799352 9842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:58.799409 9842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:58.799441 9842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:49:58.801268 9842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:58.801566 9842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:58.801954 9842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:58.807723 9847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:58.807869 9842 server_base.cc:1034] running on GCE node
W20260619 16:49:58.807785 9848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:58.807746 9850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:58.808243 9842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:58.808558 9842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:58.809720 9842 hybrid_clock.cc:648] HybridClock initialized: now 1781887798809712 us; error 36 us; skew 500 ppm
I20260619 16:49:58.811283 9842 webserver.cc:460] Webserver started at http://127.0.98.2:44733/ using document root <none> and password file <none>
I20260619 16:49:58.811573 9842 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:58.811631 9842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:58.811779 9842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:58.813072 9842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-1/data/instance:
uuid: "5d0951e8e50c44fdaa3a6cc5488455d4"
format_stamp: "Formatted at 2026-06-19 16:49:58 on dist-test-slave-jsw9"
server_key: "dfbe9b939bcedc7e1868d2da92271b6a"
server_key_iv: "496666607ebca673f59cec046a2164ca"
server_key_version: "encryptionkey@0"
I20260619 16:49:58.813442 9842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5d0951e8e50c44fdaa3a6cc5488455d4"
format_stamp: "Formatted at 2026-06-19 16:49:58 on dist-test-slave-jsw9"
server_key: "dfbe9b939bcedc7e1868d2da92271b6a"
server_key_iv: "496666607ebca673f59cec046a2164ca"
server_key_version: "encryptionkey@0"
I20260619 16:49:58.815032 9842 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 16:49:58.815934 9856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:58.816138 9842 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:58.816244 9842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-1/wal
uuid: "5d0951e8e50c44fdaa3a6cc5488455d4"
format_stamp: "Formatted at 2026-06-19 16:49:58 on dist-test-slave-jsw9"
server_key: "dfbe9b939bcedc7e1868d2da92271b6a"
server_key_iv: "496666607ebca673f59cec046a2164ca"
server_key_version: "encryptionkey@0"
I20260619 16:49:58.816316 9842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:58.864302 9842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:58.864725 9842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:58.864876 9842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:58.865176 9842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:58.865564 9842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:58.865600 9842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:58.865633 9842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:58.865656 9842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:58.872793 9842 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43805
I20260619 16:49:58.872874 9969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43805 every 8 connection(s)
I20260619 16:49:58.873271 9842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:49:58.878865 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 9842
I20260619 16:49:58.878957 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:49:58.879148 392 external_mini_cluster.cc:1428] Setting key f594b1b9b1e4f6543242f8f0b80d3140
I20260619 16:49:58.879408 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:58.874216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60045 (local address 127.0.98.62:43095)
0619 16:49:58.874276 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:58.874278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:58.874556 (+ 278us) server_negotiation.cc:407] Connection header received
0619 16:49:58.875226 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:58.875227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:58.875243 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:58.875301 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:58.876086 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:58.876694 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:58.877824 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:58.878022 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:58.878084 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:58.879087 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:58.879088 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:58.879089 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:58.879097 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:58.879113 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:58.879117 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:58.879167 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:58.879272 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 16:49:58.879319 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 16:49:58.879694 9972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:58.874179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43095 (local address 127.0.98.2:60045)
0619 16:49:58.874413 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:58.874453 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:58.875102 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:58.875386 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:58.875390 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:58.875401 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:58.875987 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:58.875993 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:58.876801 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:58.876801 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:58.877739 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:58.877744 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:58.878099 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:58.878100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:58.878281 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:58.878896 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:49:58.878896 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:58.878932 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:49:58.878968 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:58.879173 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:58.879181 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:58.879184 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:49:58.879261 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 16:49:58.879423 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
I20260619 16:49:58.880285 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43095
--builtin_ntp_servers=127.0.98.20:44027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 16:49:58.880268 9970 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43095
I20260619 16:49:58.880389 9970 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:58.880610 9970 heartbeater.cc:511] Master 127.0.98.62:43095 requested a full tablet report, sending...
I20260619 16:49:58.881134 9651 ts_manager.cc:194] Registered new tserver with Master: 5d0951e8e50c44fdaa3a6cc5488455d4 (127.0.98.2:43805)
I20260619 16:49:58.881527 9651 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:60045
W20260619 16:49:58.978823 9973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:58.979344 9973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:58.979393 9973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:58.981806 9973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:58.981911 9973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:58.981963 9973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:49:58.984840 9973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:58.985232 9973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:58.985621 9973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:58.991505 9978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:58.991505 9979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:58.991528 9981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:58.991849 9973 server_base.cc:1034] running on GCE node
I20260619 16:49:58.992100 9973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:58.992398 9973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:58.993566 9973 hybrid_clock.cc:648] HybridClock initialized: now 1781887798993535 us; error 41 us; skew 500 ppm
I20260619 16:49:58.995330 9973 webserver.cc:460] Webserver started at http://127.0.98.3:44295/ using document root <none> and password file <none>
I20260619 16:49:58.995611 9973 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:58.995666 9973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:58.995795 9973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:58.997097 9973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-2/data/instance:
uuid: "bc6eea63b4d64411a55ec37744cddbc1"
format_stamp: "Formatted at 2026-06-19 16:49:58 on dist-test-slave-jsw9"
server_key: "63f01a9e6176f9810a95a6c49fc9714d"
server_key_iv: "179732dc42c5e11cb1a184e49f865935"
server_key_version: "encryptionkey@0"
I20260619 16:49:58.997498 9973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bc6eea63b4d64411a55ec37744cddbc1"
format_stamp: "Formatted at 2026-06-19 16:49:58 on dist-test-slave-jsw9"
server_key: "63f01a9e6176f9810a95a6c49fc9714d"
server_key_iv: "179732dc42c5e11cb1a184e49f865935"
server_key_version: "encryptionkey@0"
I20260619 16:49:58.999015 9973 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 16:49:59.000643 9987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:59.000833 9973 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:49:59.000926 9973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-2/wal
uuid: "bc6eea63b4d64411a55ec37744cddbc1"
format_stamp: "Formatted at 2026-06-19 16:49:58 on dist-test-slave-jsw9"
server_key: "63f01a9e6176f9810a95a6c49fc9714d"
server_key_iv: "179732dc42c5e11cb1a184e49f865935"
server_key_version: "encryptionkey@0"
I20260619 16:49:59.000998 9973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:59.027267 9973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:59.027666 9973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:59.027792 9973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:59.028046 9973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:59.028388 9973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:59.028431 9973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:59.028461 9973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:59.028476 9973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:59.035810 9973 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34639
I20260619 16:49:59.035869 10100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34639 every 8 connection(s)
I20260619 16:49:59.036243 9973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:49:59.036691 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 9973
I20260619 16:49:59.036775 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781887741903642-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:49:59.036953 392 external_mini_cluster.cc:1428] Setting key 49da30b44b5cd3ab20bf8ceeb5e35b67
I20260619 16:49:59.042403 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:59.037219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38865 (local address 127.0.98.62:43095)
0619 16:49:59.037295 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:59.037298 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:59.037615 (+ 317us) server_negotiation.cc:407] Connection header received
0619 16:49:59.038302 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:59.038302 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:59.038313 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:59.038352 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:59.039112 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.039654 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.040696 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.040914 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.040977 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:59.041961 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:59.041962 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:59.041963 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:59.041968 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:59.041985 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:59.042001 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:59.042044 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:59.042264 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 16:49:59.042298 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 16:49:59.042888 10103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:59.037224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43095 (local address 127.0.98.3:38865)
0619 16:49:59.037463 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:59.037517 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:59.038168 (+ 651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:59.038420 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:59.038424 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:59.038434 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:59.039020 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:59.039025 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.039746 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.039747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:59.040611 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:49:59.040613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:49:59.041018 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:49:59.041019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:49:59.041182 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:49:59.041804 (+ 622us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:49:59.041804 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:59.041842 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:49:59.041873 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:59.042079 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:59.042085 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:59.042089 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:49:59.042167 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 16:49:59.042571 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":100,"threads_started":1}
I20260619 16:49:59.043686 10101 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43095
I20260619 16:49:59.043812 10101 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:59.044003 10101 heartbeater.cc:511] Master 127.0.98.62:43095 requested a full tablet report, sending...
I20260619 16:49:59.044471 9651 ts_manager.cc:194] Registered new tserver with Master: bc6eea63b4d64411a55ec37744cddbc1 (127.0.98.3:34639)
I20260619 16:49:59.044850 9651 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:38865
I20260619 16:49:59.049403 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 16:49:59.051587 9690 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:59.051039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40184 (local address 127.0.98.62:43095)
0619 16:49:59.051108 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:59.051111 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:59.051185 (+ 74us) server_negotiation.cc:407] Connection header received
0619 16:49:59.051274 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:59.051275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:59.051288 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:59.051349 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:59.051516 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40184: BlockingRecv error: recv got EOF from 127.0.0.1:40184 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 16:49:59.051968 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 9711
I20260619 16:49:59.057818 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 9842
I20260619 16:49:59.062379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 9973
I20260619 16:49:59.066736 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 9620
2026-06-19T16:49:59Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (732 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T16:49:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:49:59Z Disabled control of system clock
I20260619 16:49:59.095124 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44963
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 16:49:59.189881 10112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:59.190152 10112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:59.190178 10112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:59.191712 10112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:49:59.191787 10112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:59.191799 10112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:59.191812 10112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:49:59.191823 10112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:49:59.193514 10112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44963
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:59.193809 10112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:59.194134 10112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:59.200471 10117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:59.200482 10120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:59.200686 10112 server_base.cc:1034] running on GCE node
W20260619 16:49:59.200453 10118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:59.201054 10112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:59.201393 10112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:59.202538 10112 hybrid_clock.cc:648] HybridClock initialized: now 1781887799202554 us; error 63 us; skew 500 ppm
I20260619 16:49:59.204053 10112 webserver.cc:460] Webserver started at http://127.0.98.62:36099/ using document root <none> and password file <none>
I20260619 16:49:59.204346 10112 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:59.204399 10112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:59.204523 10112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:59.205756 10112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "00ba316c9a1440dbbaf7ed6d71bcd73d"
format_stamp: "Formatted at 2026-06-19 16:49:59 on dist-test-slave-jsw9"
server_key: "7ec3ad9b00e4e85f822d808cee947c35"
server_key_iv: "073b514baf2b65ba10042189c57238f8"
server_key_version: "encryptionkey@0"
I20260619 16:49:59.206142 10112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "00ba316c9a1440dbbaf7ed6d71bcd73d"
format_stamp: "Formatted at 2026-06-19 16:49:59 on dist-test-slave-jsw9"
server_key: "7ec3ad9b00e4e85f822d808cee947c35"
server_key_iv: "073b514baf2b65ba10042189c57238f8"
server_key_version: "encryptionkey@0"
I20260619 16:49:59.207608 10112 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:59.208734 10126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:59.209100 10112 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:59.209220 10112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "00ba316c9a1440dbbaf7ed6d71bcd73d"
format_stamp: "Formatted at 2026-06-19 16:49:59 on dist-test-slave-jsw9"
server_key: "7ec3ad9b00e4e85f822d808cee947c35"
server_key_iv: "073b514baf2b65ba10042189c57238f8"
server_key_version: "encryptionkey@0"
I20260619 16:49:59.209300 10112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:59.213304 10112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:59.213639 10112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:59.213773 10112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:59.218743 10112 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44963
I20260619 16:49:59.218829 10178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44963 every 8 connection(s)
I20260619 16:49:59.219215 10112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:49:59.220031 10179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:49:59.221112 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 10112
I20260619 16:49:59.221216 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:49:59.221377 392 external_mini_cluster.cc:1428] Setting key 54e987b12acec275a807aaa6c4be561f
I20260619 16:49:59.223838 10179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00ba316c9a1440dbbaf7ed6d71bcd73d: Bootstrap starting.
I20260619 16:49:59.224196 10182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:59.221981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43292 (local address 127.0.98.62:44963)
0619 16:49:59.222178 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:59.222224 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:59.222257 (+ 33us) server_negotiation.cc:407] Connection header received
0619 16:49:59.222758 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:59.222767 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:59.222817 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:59.222957 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:59.223724 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:59.223728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:59.223728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:59.223732 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:59.223779 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:59.223791 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:59.223852 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:59.223935 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 16:49:59.224007 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
I20260619 16:49:59.224659 10179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00ba316c9a1440dbbaf7ed6d71bcd73d: Neither blocks nor log segments found. Creating new log.
I20260619 16:49:59.224987 10179 log.cc:824] T 00000000000000000000000000000000 P 00ba316c9a1440dbbaf7ed6d71bcd73d: Log is configured to *not* fsync() on all Append() calls
I20260619 16:49:59.225710 10179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00ba316c9a1440dbbaf7ed6d71bcd73d: No bootstrap required, opened a new log
I20260619 16:49:59.227325 10179 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00ba316c9a1440dbbaf7ed6d71bcd73d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00ba316c9a1440dbbaf7ed6d71bcd73d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44963 } }
I20260619 16:49:59.227487 10179 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00ba316c9a1440dbbaf7ed6d71bcd73d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:49:59.227511 10179 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00ba316c9a1440dbbaf7ed6d71bcd73d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00ba316c9a1440dbbaf7ed6d71bcd73d, State: Initialized, Role: FOLLOWER
I20260619 16:49:59.227630 10179 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00ba316c9a1440dbbaf7ed6d71bcd73d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00ba316c9a1440dbbaf7ed6d71bcd73d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44963 } }
I20260619 16:49:59.227720 10179 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00ba316c9a1440dbbaf7ed6d71bcd73d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:49:59.227756 10179 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00ba316c9a1440dbbaf7ed6d71bcd73d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:49:59.227806 10179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00ba316c9a1440dbbaf7ed6d71bcd73d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:49:59.228421 10179 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00ba316c9a1440dbbaf7ed6d71bcd73d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00ba316c9a1440dbbaf7ed6d71bcd73d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44963 } }
I20260619 16:49:59.228521 10179 leader_election.cc:302] T 00000000000000000000000000000000 P 00ba316c9a1440dbbaf7ed6d71bcd73d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00ba316c9a1440dbbaf7ed6d71bcd73d; no voters:
I20260619 16:49:59.228699 10179 leader_election.cc:288] T 00000000000000000000000000000000 P 00ba316c9a1440dbbaf7ed6d71bcd73d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:49:59.228999 10179 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00ba316c9a1440dbbaf7ed6d71bcd73d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:49:59.229054 10184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00ba316c9a1440dbbaf7ed6d71bcd73d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:49:59.229216 10184 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00ba316c9a1440dbbaf7ed6d71bcd73d [term 1 LEADER]: Becoming Leader. State: Replica: 00ba316c9a1440dbbaf7ed6d71bcd73d, State: Running, Role: LEADER
I20260619 16:49:59.229394 10184 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00ba316c9a1440dbbaf7ed6d71bcd73d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00ba316c9a1440dbbaf7ed6d71bcd73d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44963 } }
I20260619 16:49:59.229918 10186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00ba316c9a1440dbbaf7ed6d71bcd73d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00ba316c9a1440dbbaf7ed6d71bcd73d. Latest consensus state: current_term: 1 leader_uuid: "00ba316c9a1440dbbaf7ed6d71bcd73d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00ba316c9a1440dbbaf7ed6d71bcd73d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44963 } } }
I20260619 16:49:59.230016 10186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00ba316c9a1440dbbaf7ed6d71bcd73d [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:59.230109 10185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00ba316c9a1440dbbaf7ed6d71bcd73d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00ba316c9a1440dbbaf7ed6d71bcd73d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00ba316c9a1440dbbaf7ed6d71bcd73d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44963 } } }
I20260619 16:49:59.230166 10185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00ba316c9a1440dbbaf7ed6d71bcd73d [sys.catalog]: This master's current role is: LEADER
I20260619 16:49:59.230305 10195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:49:59.231024 10195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:49:59.232539 10195 catalog_manager.cc:1347] Generated new cluster ID: 4813bd546d58412f9f97e2e0185c63e9
I20260619 16:49:59.232601 10195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:49:59.239068 10195 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:49:59.239509 10195 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:49:59.243610 10195 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00ba316c9a1440dbbaf7ed6d71bcd73d: Generated new TSK 0
I20260619 16:49:59.243875 10195 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:49:59.246039 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44963
--builtin_ntp_servers=127.0.98.20:42573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 16:49:59.348412 10203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:59.348686 10203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:59.348714 10203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:59.350132 10203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:59.350242 10203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:59.350288 10203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:49:59.352202 10203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:59.352560 10203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:59.353035 10203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:59.359651 10211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:59.359640 10208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:59.359640 10209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:59.360196 10203 server_base.cc:1034] running on GCE node
I20260619 16:49:59.360440 10203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:59.360737 10203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:59.361918 10203 hybrid_clock.cc:648] HybridClock initialized: now 1781887799361903 us; error 33 us; skew 500 ppm
I20260619 16:49:59.363785 10203 webserver.cc:460] Webserver started at http://127.0.98.1:36993/ using document root <none> and password file <none>
I20260619 16:49:59.364078 10203 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:59.364140 10203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:59.364286 10203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:59.365564 10203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-0/data/instance:
uuid: "963c9006e4aa47359ab25d9eada47cb3"
format_stamp: "Formatted at 2026-06-19 16:49:59 on dist-test-slave-jsw9"
server_key: "a1aa51ae850cd0459d3567a3b2e50700"
server_key_iv: "13b7234352e19c173ccbd8c730955ee4"
server_key_version: "encryptionkey@0"
I20260619 16:49:59.365983 10203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-0/wal/instance:
uuid: "963c9006e4aa47359ab25d9eada47cb3"
format_stamp: "Formatted at 2026-06-19 16:49:59 on dist-test-slave-jsw9"
server_key: "a1aa51ae850cd0459d3567a3b2e50700"
server_key_iv: "13b7234352e19c173ccbd8c730955ee4"
server_key_version: "encryptionkey@0"
I20260619 16:49:59.367645 10203 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260619 16:49:59.368549 10217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:59.368738 10203 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:59.368836 10203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-0/wal
uuid: "963c9006e4aa47359ab25d9eada47cb3"
format_stamp: "Formatted at 2026-06-19 16:49:59 on dist-test-slave-jsw9"
server_key: "a1aa51ae850cd0459d3567a3b2e50700"
server_key_iv: "13b7234352e19c173ccbd8c730955ee4"
server_key_version: "encryptionkey@0"
I20260619 16:49:59.368925 10203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:59.373087 10203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:59.373428 10203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:59.373570 10203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:59.373813 10203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:59.374238 10203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:59.374289 10203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:59.374330 10203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:59.374356 10203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:59.381847 10203 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37271
I20260619 16:49:59.381947 10330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37271 every 8 connection(s)
I20260619 16:49:59.382378 10203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:49:59.382855 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 10203
I20260619 16:49:59.382932 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:49:59.383131 392 external_mini_cluster.cc:1428] Setting key 8b807b84af26fa6fb71f4d8998cf2d2a
I20260619 16:49:59.384212 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44963
--builtin_ntp_servers=127.0.98.20:42573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 16:49:59.386467 10182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:59.383244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60681 (local address 127.0.98.62:44963)
0619 16:49:59.383309 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:59.383312 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:59.383921 (+ 609us) server_negotiation.cc:407] Connection header received
0619 16:49:59.384642 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:59.384643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:59.384650 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:59.384682 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:59.386176 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:59.386178 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:59.386178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:59.386209 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:59.386229 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:59.386235 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:59.386306 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:59.386365 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 16:49:59.386382 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 16:49:59.387046 10333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:59.383280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44963 (local address 127.0.98.1:60681)
0619 16:49:59.383771 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:59.383813 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:59.384520 (+ 707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:59.384793 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:59.384798 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:59.384812 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:59.386001 (+ 1189us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:49:59.386002 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:59.386038 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:49:59.386083 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:59.386305 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:59.386309 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:59.386312 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:49:59.386373 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 16:49:59.386436 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":102,"threads_started":1}
I20260619 16:49:59.387632 10331 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44963
I20260619 16:49:59.387734 10331 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:59.387898 10331 heartbeater.cc:511] Master 127.0.98.62:44963 requested a full tablet report, sending...
I20260619 16:49:59.388530 10143 ts_manager.cc:194] Registered new tserver with Master: 963c9006e4aa47359ab25d9eada47cb3 (127.0.98.1:37271)
W20260619 16:49:59.482767 10334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:59.483058 10334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:59.483085 10334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:59.484491 10334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:59.484543 10334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:59.484576 10334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:49:59.486122 10334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:59.486487 10334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:59.486838 10334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:59.494319 10339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:59.494319 10340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:59.494319 10342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:59.494357 10334 server_base.cc:1034] running on GCE node
I20260619 16:49:59.494717 10334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:59.495023 10334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:59.496232 10334 hybrid_clock.cc:648] HybridClock initialized: now 1781887799496189 us; error 37 us; skew 500 ppm
I20260619 16:49:59.497844 10334 webserver.cc:460] Webserver started at http://127.0.98.2:35281/ using document root <none> and password file <none>
I20260619 16:49:59.498087 10334 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:59.498140 10334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:59.498313 10334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:59.499624 10334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-1/data/instance:
uuid: "7635e7016ebe46a6a3cadef34c343981"
format_stamp: "Formatted at 2026-06-19 16:49:59 on dist-test-slave-jsw9"
server_key: "8731822ba5e2cda0f032b31f5e561119"
server_key_iv: "48baeacb2f86b84969af77ab9359fdf2"
server_key_version: "encryptionkey@0"
I20260619 16:49:59.499969 10334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7635e7016ebe46a6a3cadef34c343981"
format_stamp: "Formatted at 2026-06-19 16:49:59 on dist-test-slave-jsw9"
server_key: "8731822ba5e2cda0f032b31f5e561119"
server_key_iv: "48baeacb2f86b84969af77ab9359fdf2"
server_key_version: "encryptionkey@0"
I20260619 16:49:59.501412 10334 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 16:49:59.502290 10348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:59.502513 10334 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:59.502614 10334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-1/wal
uuid: "7635e7016ebe46a6a3cadef34c343981"
format_stamp: "Formatted at 2026-06-19 16:49:59 on dist-test-slave-jsw9"
server_key: "8731822ba5e2cda0f032b31f5e561119"
server_key_iv: "48baeacb2f86b84969af77ab9359fdf2"
server_key_version: "encryptionkey@0"
I20260619 16:49:59.502718 10334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:59.506577 10334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:59.506919 10334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:59.507050 10334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:59.507342 10334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:59.507756 10334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:59.507803 10334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:59.507833 10334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:59.507851 10334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:59.515550 10334 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41183
I20260619 16:49:59.515623 10461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41183 every 8 connection(s)
I20260619 16:49:59.516039 10334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:49:59.519730 10182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:59.516933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56691 (local address 127.0.98.62:44963)
0619 16:49:59.517024 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:59.517026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:59.517286 (+ 260us) server_negotiation.cc:407] Connection header received
0619 16:49:59.517898 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:59.517899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:59.517907 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:59.517978 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:59.519397 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:59.519398 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:59.519399 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:59.519404 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:59.519422 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:59.519426 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:59.519488 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:59.519635 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 16:49:59.519649 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 16:49:59.519973 10464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:59.516922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44963 (local address 127.0.98.2:56691)
0619 16:49:59.517130 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:59.517180 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:59.517774 (+ 594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:59.518050 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:59.518054 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:59.518065 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:59.519183 (+ 1118us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:49:59.519185 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:59.519222 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:49:59.519265 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:59.519515 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:59.519522 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:59.519526 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:49:59.519645 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 16:49:59.519711 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
I20260619 16:49:59.520576 10462 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44963
I20260619 16:49:59.520697 10462 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:59.520871 10462 heartbeater.cc:511] Master 127.0.98.62:44963 requested a full tablet report, sending...
I20260619 16:49:59.521282 10143 ts_manager.cc:194] Registered new tserver with Master: 7635e7016ebe46a6a3cadef34c343981 (127.0.98.2:41183)
I20260619 16:49:59.521540 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 10334
I20260619 16:49:59.521643 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:49:59.521890 392 external_mini_cluster.cc:1428] Setting key ad1ba8018fc8e78ada189935747c3b33
I20260619 16:49:59.522846 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44963
--builtin_ntp_servers=127.0.98.20:42573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 16:49:59.619150 10465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:49:59.619443 10465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:49:59.619481 10465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:49:59.621143 10465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:49:59.621212 10465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:49:59.621260 10465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:49:59.622985 10465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:49:59.623306 10465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:49:59.623658 10465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:49:59.629326 10471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:59.629333 10473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:49:59.629380 10470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:49:59.629617 10465 server_base.cc:1034] running on GCE node
I20260619 16:49:59.629894 10465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:49:59.630218 10465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:49:59.631422 10465 hybrid_clock.cc:648] HybridClock initialized: now 1781887799631407 us; error 31 us; skew 500 ppm
I20260619 16:49:59.632941 10465 webserver.cc:460] Webserver started at http://127.0.98.3:39529/ using document root <none> and password file <none>
I20260619 16:49:59.633219 10465 fs_manager.cc:359] Metadata directory not provided
I20260619 16:49:59.633267 10465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:49:59.633419 10465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:49:59.634774 10465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-2/data/instance:
uuid: "95e95d81c99344b9bba01d3126f6433f"
format_stamp: "Formatted at 2026-06-19 16:49:59 on dist-test-slave-jsw9"
server_key: "9e811fac567d2369f773a7d410f1c6c7"
server_key_iv: "79dcb2f007f5211dd4b9fb4cb529ac2d"
server_key_version: "encryptionkey@0"
I20260619 16:49:59.635150 10465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-2/wal/instance:
uuid: "95e95d81c99344b9bba01d3126f6433f"
format_stamp: "Formatted at 2026-06-19 16:49:59 on dist-test-slave-jsw9"
server_key: "9e811fac567d2369f773a7d410f1c6c7"
server_key_iv: "79dcb2f007f5211dd4b9fb4cb529ac2d"
server_key_version: "encryptionkey@0"
I20260619 16:49:59.636626 10465 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 16:49:59.637576 10479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:59.637822 10465 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:49:59.637923 10465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-2/wal
uuid: "95e95d81c99344b9bba01d3126f6433f"
format_stamp: "Formatted at 2026-06-19 16:49:59 on dist-test-slave-jsw9"
server_key: "9e811fac567d2369f773a7d410f1c6c7"
server_key_iv: "79dcb2f007f5211dd4b9fb4cb529ac2d"
server_key_version: "encryptionkey@0"
I20260619 16:49:59.638010 10465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:49:59.642122 10465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:49:59.642585 10465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:49:59.642781 10465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:49:59.643029 10465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:49:59.643406 10465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:49:59.643442 10465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:59.643465 10465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:49:59.643477 10465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:49:59.650662 10465 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38055
I20260619 16:49:59.650792 10592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38055 every 8 connection(s)
I20260619 16:49:59.651594 10465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:49:59.656059 10182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:59.652876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57531 (local address 127.0.98.62:44963)
0619 16:49:59.652967 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:59.652970 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:59.653128 (+ 158us) server_negotiation.cc:407] Connection header received
0619 16:49:59.653789 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:59.653790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:59.653799 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:59.653856 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:59.655334 (+ 1478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:49:59.655336 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:49:59.655337 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:49:59.655343 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:49:59.655473 (+ 130us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:49:59.655492 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:49:59.655785 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0619 16:49:59.655888 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 16:49:59.655954 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:49:59.656472 10595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:49:59.652611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44963 (local address 127.0.98.3:57531)
0619 16:49:59.652942 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0619 16:49:59.653001 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 16:49:59.653661 (+ 660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:49:59.653955 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:49:59.653961 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:49:59.653975 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:49:59.655127 (+ 1152us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:49:59.655128 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:49:59.655171 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:49:59.655213 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:49:59.655783 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:49:59.655794 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:49:59.655799 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:49:59.655901 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 16:49:59.656067 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":204,"threads_started":1}
I20260619 16:49:59.657191 10593 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44963
I20260619 16:49:59.657316 10593 heartbeater.cc:464] Registering TS with master...
I20260619 16:49:59.657501 10593 heartbeater.cc:511] Master 127.0.98.62:44963 requested a full tablet report, sending...
I20260619 16:49:59.658018 10143 ts_manager.cc:194] Registered new tserver with Master: 95e95d81c99344b9bba01d3126f6433f (127.0.98.3:38055)
I20260619 16:49:59.658761 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 10465
I20260619 16:49:59.659019 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781887741903642-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:49:59.659252 392 external_mini_cluster.cc:1428] Setting key b4ab35867c570943dd598dfe3adbeced
I20260619 16:49:59.660427 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 16:49:59.662886 10182 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:49:59.662126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43296 (local address 127.0.98.62:44963)
0619 16:49:59.662258 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 16:49:59.662261 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:49:59.662438 (+ 177us) server_negotiation.cc:407] Connection header received
0619 16:49:59.662478 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:49:59.662479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:49:59.662486 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:49:59.662543 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:49:59.662804 (+ 261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43296: BlockingRecv error: recv got EOF from 127.0.0.1:43296 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
I20260619 16:49:59.662915 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 10203
I20260619 16:49:59.667636 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 10334
I20260619 16:49:59.672277 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 10465
I20260619 16:49:59.676890 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 10112
2026-06-19T16:49:59Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (610 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:49:59 dist-test-slave-jsw9 krb5kdc[10603](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:49:59 dist-test-slave-jsw9 krb5kdc[10603](info): set up 2 sockets
Jun 19 16:49:59 dist-test-slave-jsw9 krb5kdc[10603](info): commencing operation
krb5kdc: starting...
W20260619 16:50:00.680809 392 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:50:00 dist-test-slave-jsw9 krb5kdc[10603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:50:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:50:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:50:00.830283 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35539
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35539
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:00.927577 10616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:00.927881 10616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:00.927919 10616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:00.929462 10616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:50:00.929539 10616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:00.929558 10616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:00.929576 10616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:50:00.929591 10616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:50:00.932055 10616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:50:00.932955 10616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:00.933558 10616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:00.940017 10621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:00.940017 10624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:00.940017 10622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:00.940264 10616 server_base.cc:1034] running on GCE node
I20260619 16:50:00.940634 10616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:00.941007 10616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:00.942237 10616 hybrid_clock.cc:648] HybridClock initialized: now 1781887800942208 us; error 47 us; skew 500 ppm
Jun 19 16:50:00 dist-test-slave-jsw9 krb5kdc[10603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:00.945817 10616 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:50:00.946492 10616 webserver.cc:460] Webserver started at http://127.0.98.62:34499/ using document root <none> and password file <none>
I20260619 16:50:00.946763 10616 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:00.946820 10616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:00.946969 10616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:00.948263 10616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "90913d6c8bf74afd812111c9a3db25e3"
format_stamp: "Formatted at 2026-06-19 16:50:00 on dist-test-slave-jsw9"
server_key: "9081046fc9f27dfb6c6620bc5bd0a201"
server_key_iv: "bd5c9649fb7ef74cb8981321b8f3b5fc"
server_key_version: "encryptionkey@0"
I20260619 16:50:00.948666 10616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "90913d6c8bf74afd812111c9a3db25e3"
format_stamp: "Formatted at 2026-06-19 16:50:00 on dist-test-slave-jsw9"
server_key: "9081046fc9f27dfb6c6620bc5bd0a201"
server_key_iv: "bd5c9649fb7ef74cb8981321b8f3b5fc"
server_key_version: "encryptionkey@0"
I20260619 16:50:00.950240 10616 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 16:50:00.951414 10631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:00.951704 10616 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:50:00.951815 10616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "90913d6c8bf74afd812111c9a3db25e3"
format_stamp: "Formatted at 2026-06-19 16:50:00 on dist-test-slave-jsw9"
server_key: "9081046fc9f27dfb6c6620bc5bd0a201"
server_key_iv: "bd5c9649fb7ef74cb8981321b8f3b5fc"
server_key_version: "encryptionkey@0"
I20260619 16:50:00.951893 10616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:00.973731 10616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:00.976807 10616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:00.977015 10616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:00.981841 10616 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35539
I20260619 16:50:00.981889 10683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35539 every 8 connection(s)
I20260619 16:50:00.982323 10616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:50:00.983443 10684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:00.986495 10684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90913d6c8bf74afd812111c9a3db25e3: Bootstrap starting.
I20260619 16:50:00.987268 10684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90913d6c8bf74afd812111c9a3db25e3: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:00.987551 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 10616
I20260619 16:50:00.987584 10684 log.cc:824] T 00000000000000000000000000000000 P 90913d6c8bf74afd812111c9a3db25e3: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:00.987656 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:50:00.987891 392 external_mini_cluster.cc:1428] Setting key baab2e45e3d857d1464c0a9671fa882b
I20260619 16:50:00.988433 10684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90913d6c8bf74afd812111c9a3db25e3: No bootstrap required, opened a new log
I20260619 16:50:00.991510 10684 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90913d6c8bf74afd812111c9a3db25e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90913d6c8bf74afd812111c9a3db25e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } }
I20260619 16:50:00.991808 10684 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90913d6c8bf74afd812111c9a3db25e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:00.991861 10684 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90913d6c8bf74afd812111c9a3db25e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90913d6c8bf74afd812111c9a3db25e3, State: Initialized, Role: FOLLOWER
I20260619 16:50:00.992020 10684 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90913d6c8bf74afd812111c9a3db25e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90913d6c8bf74afd812111c9a3db25e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } }
I20260619 16:50:00.992121 10684 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90913d6c8bf74afd812111c9a3db25e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:50:00.992161 10684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90913d6c8bf74afd812111c9a3db25e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:50:00.992247 10684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90913d6c8bf74afd812111c9a3db25e3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:00.993302 10684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90913d6c8bf74afd812111c9a3db25e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90913d6c8bf74afd812111c9a3db25e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } }
I20260619 16:50:00.993602 10684 leader_election.cc:302] T 00000000000000000000000000000000 P 90913d6c8bf74afd812111c9a3db25e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90913d6c8bf74afd812111c9a3db25e3; no voters:
I20260619 16:50:00.993887 10684 leader_election.cc:288] T 00000000000000000000000000000000 P 90913d6c8bf74afd812111c9a3db25e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:50:00.994046 10689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90913d6c8bf74afd812111c9a3db25e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:00.994284 10684 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90913d6c8bf74afd812111c9a3db25e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:50:00.994463 10689 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90913d6c8bf74afd812111c9a3db25e3 [term 1 LEADER]: Becoming Leader. State: Replica: 90913d6c8bf74afd812111c9a3db25e3, State: Running, Role: LEADER
I20260619 16:50:00.994886 10689 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90913d6c8bf74afd812111c9a3db25e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90913d6c8bf74afd812111c9a3db25e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } }
I20260619 16:50:00.995589 10691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90913d6c8bf74afd812111c9a3db25e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90913d6c8bf74afd812111c9a3db25e3. Latest consensus state: current_term: 1 leader_uuid: "90913d6c8bf74afd812111c9a3db25e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90913d6c8bf74afd812111c9a3db25e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } }
I20260619 16:50:00.995671 10691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90913d6c8bf74afd812111c9a3db25e3 [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:00.995888 10698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:50:00.996508 10698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:50:00.996846 10690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90913d6c8bf74afd812111c9a3db25e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90913d6c8bf74afd812111c9a3db25e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90913d6c8bf74afd812111c9a3db25e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } }
Jun 19 16:50:00 dist-test-slave-jsw9 krb5kdc[10603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:50:00.996965 10690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90913d6c8bf74afd812111c9a3db25e3 [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:00.999447 10698 catalog_manager.cc:1347] Generated new cluster ID: 6917bf690f2c47d8822f1c9273035869
I20260619 16:50:00.999521 10698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:50:01.002735 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:00.988482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57504 (local address 127.0.98.62:35539)
0619 16:50:00.988679 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:00.988688 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:00.988717 (+ 29us) server_negotiation.cc:407] Connection header received
0619 16:50:00.989275 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:00.989292 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:00.989351 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:00.989491 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:00.990513 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:00.991683 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:00.992926 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:00.993427 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:00.993533 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:00.997936 (+ 4403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:00.997942 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:00.997943 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:00.997960 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:01.000337 (+ 2377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:01.000803 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:01.000807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:01.000807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:01.000917 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:01.001084 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:01.001084 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:01.001085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:01.001477 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:01.001636 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:01.002317 (+ 681us) server_negotiation.cc:299] Negotiation successful
0619 16:50:01.002487 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":95,"threads_started":1}
I20260619 16:50:01.023212 10698 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:50:01.024399 10698 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:50:01.033078 10698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90913d6c8bf74afd812111c9a3db25e3: Generated new TSK 0
I20260619 16:50:01.033438 10698 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:50:01.083252 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35539
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:01.183734 10710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:01.184000 10710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:01.184034 10710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:01.185355 10710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:01.185406 10710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:01.185444 10710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:50:01.187089 10710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:50:01.187372 10710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:01.187630 10710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:01.193969 10715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:01.193981 10716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:01.194021 10710 server_base.cc:1034] running on GCE node
W20260619 16:50:01.194000 10718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:01.194520 10710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:01.194836 10710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:01.196023 10710 hybrid_clock.cc:648] HybridClock initialized: now 1781887801196005 us; error 48 us; skew 500 ppm
Jun 19 16:50:01 dist-test-slave-jsw9 krb5kdc[10603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:01.203913 10710 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:50:01.204556 10710 webserver.cc:460] Webserver started at http://127.0.98.1:34321/ using document root <none> and password file <none>
I20260619 16:50:01.204809 10710 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:01.204862 10710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:01.205010 10710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:01.206364 10710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-0/data/instance:
uuid: "0f9f874691a94ad886e005e4df690c8b"
format_stamp: "Formatted at 2026-06-19 16:50:01 on dist-test-slave-jsw9"
server_key: "ca345fd07d8090098e221a1fe607fa14"
server_key_iv: "ccd537199cec01d5d0e77f39dcf30b4e"
server_key_version: "encryptionkey@0"
I20260619 16:50:01.206753 10710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0f9f874691a94ad886e005e4df690c8b"
format_stamp: "Formatted at 2026-06-19 16:50:01 on dist-test-slave-jsw9"
server_key: "ca345fd07d8090098e221a1fe607fa14"
server_key_iv: "ccd537199cec01d5d0e77f39dcf30b4e"
server_key_version: "encryptionkey@0"
I20260619 16:50:01.208293 10710 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 16:50:01.209156 10725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:01.209329 10710 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:50:01.209421 10710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-0/wal
uuid: "0f9f874691a94ad886e005e4df690c8b"
format_stamp: "Formatted at 2026-06-19 16:50:01 on dist-test-slave-jsw9"
server_key: "ca345fd07d8090098e221a1fe607fa14"
server_key_iv: "ccd537199cec01d5d0e77f39dcf30b4e"
server_key_version: "encryptionkey@0"
I20260619 16:50:01.209488 10710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:01.226403 10710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:01.228945 10710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:01.229112 10710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:01.229391 10710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:50:01.229743 10710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:50:01.229779 10710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:01.229832 10710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:50:01.229857 10710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:01.237402 10710 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35671
I20260619 16:50:01.237488 10838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35671 every 8 connection(s)
I20260619 16:50:01.237905 10710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:50:01.240996 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 10710
I20260619 16:50:01.241109 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:50:01.241331 392 external_mini_cluster.cc:1428] Setting key e01e75fa57aaba23a4083035cc2dd03e
Jun 19 16:50:01 dist-test-slave-jsw9 krb5kdc[10603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 16:50:01.251217 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:01.238854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43815 (local address 127.0.98.62:35539)
0619 16:50:01.238977 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:01.238979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:01.239234 (+ 255us) server_negotiation.cc:407] Connection header received
0619 16:50:01.239806 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:01.239807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:01.239820 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:01.239887 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:01.241157 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:01.241901 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:01.243358 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:01.243669 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:01.243750 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:01.246856 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:01.246859 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:01.246860 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:01.246873 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:01.248777 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:01.249564 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:01.249567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:01.249567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:01.249668 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:01.250072 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:01.250073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:01.250074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:01.250299 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:01.250387 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:01.250960 (+ 573us) server_negotiation.cc:299] Negotiation successful
0619 16:50:01.251083 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 16:50:01.251436 10841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:01.238796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.1:43815)
0619 16:50:01.238980 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:01.239031 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:01.239649 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:01.240003 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:01.240008 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:01.240693 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:01.241001 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:01.241008 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:01.242026 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:01.242028 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:01.243238 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:01.243243 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:01.243811 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:01.243822 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:01.244005 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:01.244699 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:01.244714 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:01.246709 (+ 1995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:01.248980 (+ 2271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:01.249001 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:01.249006 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:01.249426 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:01.249793 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:01.249795 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:01.249796 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:01.249988 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:01.250563 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:01.250574 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:01.250782 (+ 208us) client_negotiation.cc:769] Sending connection context
0619 16:50:01.250910 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 16:50:01.251129 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":93,"threads_started":1}
I20260619 16:50:01.252498 10839 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35539
I20260619 16:50:01.252664 10839 heartbeater.cc:464] Registering TS with master...
I20260619 16:50:01.252929 10839 heartbeater.cc:511] Master 127.0.98.62:35539 requested a full tablet report, sending...
I20260619 16:50:01.253716 10648 ts_manager.cc:194] Registered new tserver with Master: 0f9f874691a94ad886e005e4df690c8b (127.0.98.1:35671)
I20260619 16:50:01.254531 10648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:50:01.274354 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35539
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:01.376369 10844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:01.376686 10844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:01.376721 10844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:01.378262 10844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:01.378334 10844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:01.378371 10844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:50:01.380214 10844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:50:01.380517 10844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:01.380820 10844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:01.386586 10850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:01.386579 10852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:01.386579 10849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:01.386952 10844 server_base.cc:1034] running on GCE node
I20260619 16:50:01.387197 10844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:01.387482 10844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:01.388669 10844 hybrid_clock.cc:648] HybridClock initialized: now 1781887801388650 us; error 36 us; skew 500 ppm
Jun 19 16:50:01 dist-test-slave-jsw9 krb5kdc[10603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:01.392261 10844 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:50:01.392956 10844 webserver.cc:460] Webserver started at http://127.0.98.2:38967/ using document root <none> and password file <none>
I20260619 16:50:01.393234 10844 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:01.393294 10844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:01.393446 10844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:01.394915 10844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-1/data/instance:
uuid: "9c12a35728fe4bfc98fb41abbad60130"
format_stamp: "Formatted at 2026-06-19 16:50:01 on dist-test-slave-jsw9"
server_key: "de3895b2d3b35f2058e1157b95f19db6"
server_key_iv: "a18bc27bf548d7d702276b99267601b8"
server_key_version: "encryptionkey@0"
I20260619 16:50:01.395351 10844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9c12a35728fe4bfc98fb41abbad60130"
format_stamp: "Formatted at 2026-06-19 16:50:01 on dist-test-slave-jsw9"
server_key: "de3895b2d3b35f2058e1157b95f19db6"
server_key_iv: "a18bc27bf548d7d702276b99267601b8"
server_key_version: "encryptionkey@0"
I20260619 16:50:01.397163 10844 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:50:01.398232 10859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:01.398428 10844 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:50:01.398533 10844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-1/wal
uuid: "9c12a35728fe4bfc98fb41abbad60130"
format_stamp: "Formatted at 2026-06-19 16:50:01 on dist-test-slave-jsw9"
server_key: "de3895b2d3b35f2058e1157b95f19db6"
server_key_iv: "a18bc27bf548d7d702276b99267601b8"
server_key_version: "encryptionkey@0"
I20260619 16:50:01.398607 10844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:01.409667 10844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:01.412525 10844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:01.412688 10844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:01.413031 10844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:50:01.413415 10844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:50:01.413460 10844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:01.413488 10844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:50:01.413499 10844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:01.421083 10844 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45455
I20260619 16:50:01.421257 10972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45455 every 8 connection(s)
I20260619 16:50:01.421579 10844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 16:50:01 dist-test-slave-jsw9 krb5kdc[10603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:50:01.431846 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 10844
I20260619 16:50:01.431967 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:50:01.432161 392 external_mini_cluster.cc:1428] Setting key f412bf98f999750a72cb3f51bfdbb79c
I20260619 16:50:01.433987 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:01.422543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48605 (local address 127.0.98.62:35539)
0619 16:50:01.422621 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:01.422624 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:01.422891 (+ 267us) server_negotiation.cc:407] Connection header received
0619 16:50:01.423648 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:01.423649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:01.423668 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:01.423728 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:01.424969 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:01.425545 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:01.426766 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:01.427061 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:01.427141 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:01.430006 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:01.430009 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:01.430010 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:01.430020 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:01.431931 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:01.432613 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:01.432615 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:01.432616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:01.432686 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:01.433095 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:01.433096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:01.433096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:01.433251 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:01.433323 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:01.433766 (+ 443us) server_negotiation.cc:299] Negotiation successful
0619 16:50:01.433879 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 16:50:01.434265 10975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:01.422500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.2:48605)
0619 16:50:01.422684 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:01.422734 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:01.423469 (+ 735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:01.423870 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:01.423875 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:01.424599 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:01.424861 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:01.424867 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:01.425652 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:01.425653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:01.426625 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:01.426627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:01.427189 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:01.427190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:01.427344 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:01.428055 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:01.428067 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:01.429868 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:01.432135 (+ 2267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:01.432149 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:01.432152 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:01.432534 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:01.432751 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:01.432752 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:01.432752 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:01.433020 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:01.433325 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:01.433334 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:01.433591 (+ 257us) client_negotiation.cc:769] Sending connection context
0619 16:50:01.433725 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 16:50:01.433945 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":93,"threads_started":1}
I20260619 16:50:01.434795 10973 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35539
I20260619 16:50:01.434930 10973 heartbeater.cc:464] Registering TS with master...
I20260619 16:50:01.435200 10973 heartbeater.cc:511] Master 127.0.98.62:35539 requested a full tablet report, sending...
I20260619 16:50:01.435678 10648 ts_manager.cc:194] Registered new tserver with Master: 9c12a35728fe4bfc98fb41abbad60130 (127.0.98.2:45455)
I20260619 16:50:01.436306 10648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48605
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:50:01.471414 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35539
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:01.578898 10978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:01.579234 10978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:01.579270 10978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:01.581020 10978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:01.581097 10978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:01.581135 10978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:50:01.583259 10978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:50:01.583592 10978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:01.583895 10978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:01.589922 10986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:01.590101 10978 server_base.cc:1034] running on GCE node
W20260619 16:50:01.589921 10984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:01.589928 10983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:01.590461 10978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:01.590781 10978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:01.591991 10978 hybrid_clock.cc:648] HybridClock initialized: now 1781887801591956 us; error 61 us; skew 500 ppm
Jun 19 16:50:01 dist-test-slave-jsw9 krb5kdc[10603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:01.596844 10978 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:50:01.597472 10978 webserver.cc:460] Webserver started at http://127.0.98.3:42433/ using document root <none> and password file <none>
I20260619 16:50:01.597862 10978 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:01.597937 10978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:01.598090 10978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:50:01.599529 10978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-2/data/instance:
uuid: "c07e63c69fe244b3accacdca2c7489ec"
format_stamp: "Formatted at 2026-06-19 16:50:01 on dist-test-slave-jsw9"
server_key: "96709b7bf1c45d3e8ad7ca06715a5054"
server_key_iv: "5e8e9dde87d3ae103cf8e81a52ef3ec7"
server_key_version: "encryptionkey@0"
I20260619 16:50:01.600407 10978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c07e63c69fe244b3accacdca2c7489ec"
format_stamp: "Formatted at 2026-06-19 16:50:01 on dist-test-slave-jsw9"
server_key: "96709b7bf1c45d3e8ad7ca06715a5054"
server_key_iv: "5e8e9dde87d3ae103cf8e81a52ef3ec7"
server_key_version: "encryptionkey@0"
I20260619 16:50:01.602347 10978 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 16:50:01.603495 10993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:01.604038 10978 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:50:01.604158 10978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-2/wal
uuid: "c07e63c69fe244b3accacdca2c7489ec"
format_stamp: "Formatted at 2026-06-19 16:50:01 on dist-test-slave-jsw9"
server_key: "96709b7bf1c45d3e8ad7ca06715a5054"
server_key_iv: "5e8e9dde87d3ae103cf8e81a52ef3ec7"
server_key_version: "encryptionkey@0"
I20260619 16:50:01.604255 10978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:01.612478 10978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:01.616351 10978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:01.616570 10978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:01.617000 10978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:50:01.617431 10978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:50:01.617473 10978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:01.617507 10978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:50:01.617525 10978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:01.625275 10978 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45925
I20260619 16:50:01.625342 11106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45925 every 8 connection(s)
I20260619 16:50:01.625720 10978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:50:01.629002 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 10978
I20260619 16:50:01.629099 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:50:01.629282 392 external_mini_cluster.cc:1428] Setting key bc5ab151dbee7714a0fde02c5b707a7e
Jun 19 16:50:01 dist-test-slave-jsw9 krb5kdc[10603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:50:01.639878 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:01.626715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45479 (local address 127.0.98.62:35539)
0619 16:50:01.626865 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:01.626867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:01.627037 (+ 170us) server_negotiation.cc:407] Connection header received
0619 16:50:01.627856 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:01.627858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:01.627873 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:01.627929 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:01.629305 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:01.630034 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:01.631296 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:01.631901 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:01.631995 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:01.636274 (+ 4279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:01.636277 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:01.636277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:01.636288 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:01.638039 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:01.638687 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:01.638689 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:01.638689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:01.638753 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:01.639031 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:01.639032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:01.639032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:01.639182 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:01.639244 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:01.639586 (+ 342us) server_negotiation.cc:299] Negotiation successful
0619 16:50:01.639761 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260619 16:50:01.640204 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:01.626651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.3:45479)
0619 16:50:01.626864 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:01.626919 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:01.627675 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:01.628060 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:01.628065 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:01.628764 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:01.629090 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:01.629098 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:01.630153 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:01.630154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:01.631184 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:01.631187 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:01.632025 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:01.632026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:01.632190 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:01.633091 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:01.633109 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:01.636132 (+ 3023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:01.638254 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:01.638268 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:01.638272 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:01.638593 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:01.638817 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:01.638818 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:01.638819 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:01.638957 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:01.639243 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:01.639252 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:01.639439 (+ 187us) client_negotiation.cc:769] Sending connection context
0619 16:50:01.639537 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 16:50:01.639908 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":107,"threads_started":1}
I20260619 16:50:01.641311 11107 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35539
I20260619 16:50:01.641453 11107 heartbeater.cc:464] Registering TS with master...
I20260619 16:50:01.641723 11107 heartbeater.cc:511] Master 127.0.98.62:35539 requested a full tablet report, sending...
I20260619 16:50:01.642289 10648 ts_manager.cc:194] Registered new tserver with Master: c07e63c69fe244b3accacdca2c7489ec (127.0.98.3:45925)
I20260619 16:50:01.642757 10648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45479
I20260619 16:50:01.653460 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:50:01.663859 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:01.655719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57520 (local address 127.0.98.62:35539)
0619 16:50:01.655778 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:01.655780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:01.655790 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:50:01.655820 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:01.655821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:01.655832 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:01.655889 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:01.656704 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:01.657380 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:01.658836 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:01.659055 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:01.659130 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:01.660748 (+ 1618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:01.660750 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:01.660750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:01.660761 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:01.662403 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:01.662884 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:01.662885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:01.662886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:01.662950 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:01.663108 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:01.663108 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:01.663108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:01.663320 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:01.663406 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:01.663643 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 16:50:01.663758 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 16:50:01.666409 10648 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57520:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:50:01.669864 10648 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:50:01.679380 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:01.673279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45925 (local address 127.0.0.1:47990)
0619 16:50:01.673687 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:01.673746 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:01.673836 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:01.674285 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:01.674289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:01.674292 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:01.674420 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:01.674423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:01.675713 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:01.675714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:01.677959 (+ 2245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:01.677963 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:01.678896 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:01.678898 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:01.679053 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:01.679080 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 16:50:01.679167 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 16:50:01.679248 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":404,"thread_start_us":101,"threads_started":1}
I20260619 16:50:01.679847 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:01.673527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45455 (local address 127.0.0.1:57850)
0619 16:50:01.673971 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:01.673979 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:01.674355 (+ 376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:01.674671 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:01.674672 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:01.674674 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:01.674740 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:01.674742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:01.676236 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:01.676238 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:01.678079 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:01.678082 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:01.679576 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:01.679577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:01.679654 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:01.679659 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:50:01.679710 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 16:50:01.679765 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":301,"thread_start_us":119,"threads_started":1}
I20260619 16:50:01.680052 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:01.673723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47990 (local address 127.0.98.3:45925)
0619 16:50:01.673904 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:01.673908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:01.673921 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:50:01.673971 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:01.673974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:01.674126 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:01.674635 (+ 509us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:01.674736 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:01.675604 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:01.678094 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:01.678800 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:01.678889 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:01.678982 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:01.679200 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 16:50:01.679258 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":87,"threads_started":1}
I20260619 16:50:01.680058 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:01.673405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33392 (local address 127.0.98.1:35671)
0619 16:50:01.673636 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:01.673643 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:01.674242 (+ 599us) server_negotiation.cc:407] Connection header received
0619 16:50:01.674390 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:01.674392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:01.674409 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:01.674471 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:01.674939 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:01.676201 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:01.678082 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:01.679249 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:01.679335 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:01.679616 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:01.679824 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 16:50:01.679883 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":101,"threads_started":1}
I20260619 16:50:01.680184 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:01.673765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57850 (local address 127.0.98.2:45455)
0619 16:50:01.674519 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:01.674524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:01.674534 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:50:01.674576 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:01.674578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:01.674594 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:01.674674 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:01.674802 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:01.676016 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:01.678306 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:01.679298 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:01.679456 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:01.679531 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:01.680032 (+ 501us) server_negotiation.cc:299] Negotiation successful
0619 16:50:01.680097 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":113,"threads_started":1}
I20260619 16:50:01.680341 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:01.673279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35671 (local address 127.0.0.1:33392)
0619 16:50:01.673912 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:01.673923 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:01.674277 (+ 354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:01.674759 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:01.674760 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:01.674763 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:01.674812 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:01.674813 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:01.676325 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:01.676326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:01.677959 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:01.677963 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:01.679649 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:01.679650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:01.679732 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:01.679736 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:50:01.680212 (+ 476us) client_negotiation.cc:240] Negotiation successful
0619 16:50:01.680268 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"mutex_wait_us":32,"thread_start_us":87,"threads_started":1}
I20260619 16:50:01.680899 10907 tablet_service.cc:1467] Processing CreateTablet for tablet 74f16f22ef9c4ef98271b101d33b5636 (DEFAULT_TABLE table=test-table [id=ec942b46b98e48698b4b3884822a9d81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:50:01.680967 11041 tablet_service.cc:1467] Processing CreateTablet for tablet 74f16f22ef9c4ef98271b101d33b5636 (DEFAULT_TABLE table=test-table [id=ec942b46b98e48698b4b3884822a9d81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:50:01.681017 10773 tablet_service.cc:1467] Processing CreateTablet for tablet 74f16f22ef9c4ef98271b101d33b5636 (DEFAULT_TABLE table=test-table [id=ec942b46b98e48698b4b3884822a9d81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:50:01.681350 10907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74f16f22ef9c4ef98271b101d33b5636. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:01.681349 10773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74f16f22ef9c4ef98271b101d33b5636. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:01.681349 11041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74f16f22ef9c4ef98271b101d33b5636. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:01.685029 11124 tablet_bootstrap.cc:492] T 74f16f22ef9c4ef98271b101d33b5636 P 9c12a35728fe4bfc98fb41abbad60130: Bootstrap starting.
I20260619 16:50:01.685515 11125 tablet_bootstrap.cc:492] T 74f16f22ef9c4ef98271b101d33b5636 P c07e63c69fe244b3accacdca2c7489ec: Bootstrap starting.
I20260619 16:50:01.685989 11124 tablet_bootstrap.cc:654] T 74f16f22ef9c4ef98271b101d33b5636 P 9c12a35728fe4bfc98fb41abbad60130: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:01.686523 11125 tablet_bootstrap.cc:654] T 74f16f22ef9c4ef98271b101d33b5636 P c07e63c69fe244b3accacdca2c7489ec: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:01.686553 11124 log.cc:824] T 74f16f22ef9c4ef98271b101d33b5636 P 9c12a35728fe4bfc98fb41abbad60130: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:01.686936 11126 tablet_bootstrap.cc:492] T 74f16f22ef9c4ef98271b101d33b5636 P 0f9f874691a94ad886e005e4df690c8b: Bootstrap starting.
I20260619 16:50:01.687121 11125 log.cc:824] T 74f16f22ef9c4ef98271b101d33b5636 P c07e63c69fe244b3accacdca2c7489ec: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:01.687531 11124 tablet_bootstrap.cc:492] T 74f16f22ef9c4ef98271b101d33b5636 P 9c12a35728fe4bfc98fb41abbad60130: No bootstrap required, opened a new log
I20260619 16:50:01.687619 11124 ts_tablet_manager.cc:1397] T 74f16f22ef9c4ef98271b101d33b5636 P 9c12a35728fe4bfc98fb41abbad60130: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:50:01.687681 11126 tablet_bootstrap.cc:654] T 74f16f22ef9c4ef98271b101d33b5636 P 0f9f874691a94ad886e005e4df690c8b: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:01.687929 11125 tablet_bootstrap.cc:492] T 74f16f22ef9c4ef98271b101d33b5636 P c07e63c69fe244b3accacdca2c7489ec: No bootstrap required, opened a new log
I20260619 16:50:01.687995 11125 ts_tablet_manager.cc:1397] T 74f16f22ef9c4ef98271b101d33b5636 P c07e63c69fe244b3accacdca2c7489ec: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 16:50:01.688279 11126 log.cc:824] T 74f16f22ef9c4ef98271b101d33b5636 P 0f9f874691a94ad886e005e4df690c8b: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:01.689263 11126 tablet_bootstrap.cc:492] T 74f16f22ef9c4ef98271b101d33b5636 P 0f9f874691a94ad886e005e4df690c8b: No bootstrap required, opened a new log
I20260619 16:50:01.689406 11126 ts_tablet_manager.cc:1397] T 74f16f22ef9c4ef98271b101d33b5636 P 0f9f874691a94ad886e005e4df690c8b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:50:01.689992 11125 raft_consensus.cc:348] T 74f16f22ef9c4ef98271b101d33b5636 P c07e63c69fe244b3accacdca2c7489ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07e63c69fe244b3accacdca2c7489ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45925 } } peers { permanent_uuid: "9c12a35728fe4bfc98fb41abbad60130" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45455 } } peers { permanent_uuid: "0f9f874691a94ad886e005e4df690c8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } }
I20260619 16:50:01.690178 11125 raft_consensus.cc:374] T 74f16f22ef9c4ef98271b101d33b5636 P c07e63c69fe244b3accacdca2c7489ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:01.690281 11125 raft_consensus.cc:729] T 74f16f22ef9c4ef98271b101d33b5636 P c07e63c69fe244b3accacdca2c7489ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c07e63c69fe244b3accacdca2c7489ec, State: Initialized, Role: FOLLOWER
I20260619 16:50:01.690236 11124 raft_consensus.cc:348] T 74f16f22ef9c4ef98271b101d33b5636 P 9c12a35728fe4bfc98fb41abbad60130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07e63c69fe244b3accacdca2c7489ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45925 } } peers { permanent_uuid: "9c12a35728fe4bfc98fb41abbad60130" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45455 } } peers { permanent_uuid: "0f9f874691a94ad886e005e4df690c8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } }
I20260619 16:50:01.690445 11124 raft_consensus.cc:374] T 74f16f22ef9c4ef98271b101d33b5636 P 9c12a35728fe4bfc98fb41abbad60130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:01.690443 11125 consensus_queue.cc:260] T 74f16f22ef9c4ef98271b101d33b5636 P c07e63c69fe244b3accacdca2c7489ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07e63c69fe244b3accacdca2c7489ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45925 } } peers { permanent_uuid: "9c12a35728fe4bfc98fb41abbad60130" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45455 } } peers { permanent_uuid: "0f9f874691a94ad886e005e4df690c8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } }
I20260619 16:50:01.690523 11124 raft_consensus.cc:729] T 74f16f22ef9c4ef98271b101d33b5636 P 9c12a35728fe4bfc98fb41abbad60130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c12a35728fe4bfc98fb41abbad60130, State: Initialized, Role: FOLLOWER
I20260619 16:50:01.690670 11124 consensus_queue.cc:260] T 74f16f22ef9c4ef98271b101d33b5636 P 9c12a35728fe4bfc98fb41abbad60130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07e63c69fe244b3accacdca2c7489ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45925 } } peers { permanent_uuid: "9c12a35728fe4bfc98fb41abbad60130" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45455 } } peers { permanent_uuid: "0f9f874691a94ad886e005e4df690c8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } }
I20260619 16:50:01.690804 11125 ts_tablet_manager.cc:1428] T 74f16f22ef9c4ef98271b101d33b5636 P c07e63c69fe244b3accacdca2c7489ec: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 16:50:01.690949 11107 heartbeater.cc:503] Master 127.0.98.62:35539 was elected leader, sending a full tablet report...
I20260619 16:50:01.691026 10973 heartbeater.cc:503] Master 127.0.98.62:35539 was elected leader, sending a full tablet report...
I20260619 16:50:01.691352 11124 ts_tablet_manager.cc:1428] T 74f16f22ef9c4ef98271b101d33b5636 P 9c12a35728fe4bfc98fb41abbad60130: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260619 16:50:01.691824 11126 raft_consensus.cc:348] T 74f16f22ef9c4ef98271b101d33b5636 P 0f9f874691a94ad886e005e4df690c8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07e63c69fe244b3accacdca2c7489ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45925 } } peers { permanent_uuid: "9c12a35728fe4bfc98fb41abbad60130" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45455 } } peers { permanent_uuid: "0f9f874691a94ad886e005e4df690c8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } }
I20260619 16:50:01.692034 11126 raft_consensus.cc:374] T 74f16f22ef9c4ef98271b101d33b5636 P 0f9f874691a94ad886e005e4df690c8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:01.692121 11126 raft_consensus.cc:729] T 74f16f22ef9c4ef98271b101d33b5636 P 0f9f874691a94ad886e005e4df690c8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f9f874691a94ad886e005e4df690c8b, State: Initialized, Role: FOLLOWER
I20260619 16:50:01.692335 11126 consensus_queue.cc:260] T 74f16f22ef9c4ef98271b101d33b5636 P 0f9f874691a94ad886e005e4df690c8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07e63c69fe244b3accacdca2c7489ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45925 } } peers { permanent_uuid: "9c12a35728fe4bfc98fb41abbad60130" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45455 } } peers { permanent_uuid: "0f9f874691a94ad886e005e4df690c8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } }
I20260619 16:50:01.692703 10839 heartbeater.cc:503] Master 127.0.98.62:35539 was elected leader, sending a full tablet report...
I20260619 16:50:01.692708 11126 ts_tablet_manager.cc:1428] T 74f16f22ef9c4ef98271b101d33b5636 P 0f9f874691a94ad886e005e4df690c8b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260619 16:50:01.738873 10840 tablet.cc:2369] T 74f16f22ef9c4ef98271b101d33b5636 P 0f9f874691a94ad886e005e4df690c8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:50:01.790663 11132 raft_consensus.cc:482] T 74f16f22ef9c4ef98271b101d33b5636 P 0f9f874691a94ad886e005e4df690c8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:50:01.791064 11132 raft_consensus.cc:504] T 74f16f22ef9c4ef98271b101d33b5636 P 0f9f874691a94ad886e005e4df690c8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07e63c69fe244b3accacdca2c7489ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45925 } } peers { permanent_uuid: "9c12a35728fe4bfc98fb41abbad60130" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45455 } } peers { permanent_uuid: "0f9f874691a94ad886e005e4df690c8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } }
I20260619 16:50:01.791731 11132 leader_election.cc:288] T 74f16f22ef9c4ef98271b101d33b5636 P 0f9f874691a94ad886e005e4df690c8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c07e63c69fe244b3accacdca2c7489ec (127.0.98.3:45925), 9c12a35728fe4bfc98fb41abbad60130 (127.0.98.2:45455)
I20260619 16:50:01.795863 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:01.792181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59679 (local address 127.0.98.3:45925)
0619 16:50:01.792446 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:01.792448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:01.792541 (+ 93us) server_negotiation.cc:407] Connection header received
0619 16:50:01.792614 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:01.792615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:01.792630 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:01.792692 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:01.792896 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:01.793395 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:01.794906 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:01.795475 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:01.795538 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:01.795552 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:01.795730 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 16:50:01.795781 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 16:50:01.795957 11133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:01.792043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45925 (local address 127.0.98.1:59679)
0619 16:50:01.792300 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:01.792369 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:01.792548 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:01.792694 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:01.792695 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:01.792697 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:01.792824 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:01.792826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:01.793643 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:01.793645 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:01.794684 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:01.794687 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:01.795560 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:01.795561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:01.795638 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:01.795642 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:50:01.795686 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 16:50:01.795745 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":2,"thread_start_us":133,"threads_started":1}
I20260619 16:50:01.796064 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:01.792402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53373 (local address 127.0.98.2:45455)
0619 16:50:01.792469 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:01.792471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:01.792490 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:50:01.792586 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:01.792587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:01.792600 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:01.792646 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:01.792827 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:01.793353 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:01.794815 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:01.795432 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:01.795509 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:01.795525 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:01.795663 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 16:50:01.795753 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 16:50:01.796223 11134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:01.792298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45455 (local address 127.0.98.1:53373)
0619 16:50:01.792430 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:01.792440 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:01.792511 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:01.792654 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:01.792655 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:01.792658 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:01.792757 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:01.792760 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:01.793445 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:01.793446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:01.794643 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:01.794647 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:01.795514 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:01.795515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:01.795597 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:01.795601 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:50:01.795650 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 16:50:01.795748 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":100,"thread_start_us":55,"threads_started":1}
I20260619 16:50:01.796480 11061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f16f22ef9c4ef98271b101d33b5636" candidate_uuid: "0f9f874691a94ad886e005e4df690c8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c07e63c69fe244b3accacdca2c7489ec" is_pre_election: true
I20260619 16:50:01.796690 11061 raft_consensus.cc:2454] T 74f16f22ef9c4ef98271b101d33b5636 P c07e63c69fe244b3accacdca2c7489ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f9f874691a94ad886e005e4df690c8b in term 0.
I20260619 16:50:01.796628 10927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f16f22ef9c4ef98271b101d33b5636" candidate_uuid: "0f9f874691a94ad886e005e4df690c8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c12a35728fe4bfc98fb41abbad60130" is_pre_election: true
I20260619 16:50:01.796871 10927 raft_consensus.cc:2454] T 74f16f22ef9c4ef98271b101d33b5636 P 9c12a35728fe4bfc98fb41abbad60130 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f9f874691a94ad886e005e4df690c8b in term 0.
I20260619 16:50:01.797016 10727 leader_election.cc:302] T 74f16f22ef9c4ef98271b101d33b5636 P 0f9f874691a94ad886e005e4df690c8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f9f874691a94ad886e005e4df690c8b, c07e63c69fe244b3accacdca2c7489ec; no voters:
I20260619 16:50:01.797168 11132 raft_consensus.cc:2781] T 74f16f22ef9c4ef98271b101d33b5636 P 0f9f874691a94ad886e005e4df690c8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:50:01.797236 11132 raft_consensus.cc:482] T 74f16f22ef9c4ef98271b101d33b5636 P 0f9f874691a94ad886e005e4df690c8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:50:01.797257 11132 raft_consensus.cc:3037] T 74f16f22ef9c4ef98271b101d33b5636 P 0f9f874691a94ad886e005e4df690c8b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:01.798720 11132 raft_consensus.cc:504] T 74f16f22ef9c4ef98271b101d33b5636 P 0f9f874691a94ad886e005e4df690c8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07e63c69fe244b3accacdca2c7489ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45925 } } peers { permanent_uuid: "9c12a35728fe4bfc98fb41abbad60130" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45455 } } peers { permanent_uuid: "0f9f874691a94ad886e005e4df690c8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } }
I20260619 16:50:01.798923 11132 leader_election.cc:288] T 74f16f22ef9c4ef98271b101d33b5636 P 0f9f874691a94ad886e005e4df690c8b [CANDIDATE]: Term 1 election: Requested vote from peers c07e63c69fe244b3accacdca2c7489ec (127.0.98.3:45925), 9c12a35728fe4bfc98fb41abbad60130 (127.0.98.2:45455)
I20260619 16:50:01.799181 11061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f16f22ef9c4ef98271b101d33b5636" candidate_uuid: "0f9f874691a94ad886e005e4df690c8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c07e63c69fe244b3accacdca2c7489ec"
I20260619 16:50:01.799266 10927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f16f22ef9c4ef98271b101d33b5636" candidate_uuid: "0f9f874691a94ad886e005e4df690c8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c12a35728fe4bfc98fb41abbad60130"
I20260619 16:50:01.799311 11061 raft_consensus.cc:3037] T 74f16f22ef9c4ef98271b101d33b5636 P c07e63c69fe244b3accacdca2c7489ec [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:01.799381 10927 raft_consensus.cc:3037] T 74f16f22ef9c4ef98271b101d33b5636 P 9c12a35728fe4bfc98fb41abbad60130 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:01.800658 11061 raft_consensus.cc:2454] T 74f16f22ef9c4ef98271b101d33b5636 P c07e63c69fe244b3accacdca2c7489ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f9f874691a94ad886e005e4df690c8b in term 1.
I20260619 16:50:01.800706 10927 raft_consensus.cc:2454] T 74f16f22ef9c4ef98271b101d33b5636 P 9c12a35728fe4bfc98fb41abbad60130 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f9f874691a94ad886e005e4df690c8b in term 1.
I20260619 16:50:01.800972 10727 leader_election.cc:302] T 74f16f22ef9c4ef98271b101d33b5636 P 0f9f874691a94ad886e005e4df690c8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f9f874691a94ad886e005e4df690c8b, c07e63c69fe244b3accacdca2c7489ec; no voters:
I20260619 16:50:01.801074 11132 raft_consensus.cc:2781] T 74f16f22ef9c4ef98271b101d33b5636 P 0f9f874691a94ad886e005e4df690c8b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:01.801293 11132 raft_consensus.cc:686] T 74f16f22ef9c4ef98271b101d33b5636 P 0f9f874691a94ad886e005e4df690c8b [term 1 LEADER]: Becoming Leader. State: Replica: 0f9f874691a94ad886e005e4df690c8b, State: Running, Role: LEADER
I20260619 16:50:01.801433 11132 consensus_queue.cc:237] T 74f16f22ef9c4ef98271b101d33b5636 P 0f9f874691a94ad886e005e4df690c8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07e63c69fe244b3accacdca2c7489ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45925 } } peers { permanent_uuid: "9c12a35728fe4bfc98fb41abbad60130" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45455 } } peers { permanent_uuid: "0f9f874691a94ad886e005e4df690c8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } }
I20260619 16:50:01.802855 10648 catalog_manager.cc:5495] T 74f16f22ef9c4ef98271b101d33b5636 P 0f9f874691a94ad886e005e4df690c8b reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f9f874691a94ad886e005e4df690c8b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0f9f874691a94ad886e005e4df690c8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07e63c69fe244b3accacdca2c7489ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c12a35728fe4bfc98fb41abbad60130" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f9f874691a94ad886e005e4df690c8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35671 } health_report { overall_health: HEALTHY } } }
I20260619 16:50:01.832453 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:01.829549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33408 (local address 127.0.98.1:35671)
0619 16:50:01.829622 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:01.829625 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:01.829633 (+ 8us) server_negotiation.cc:407] Connection header received
0619 16:50:01.829690 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:01.829691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:01.829727 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:01.829806 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:50:01.829967 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:01.830552 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:01.831361 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:01.831759 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:01.831849 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:01.832053 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:50:01.832208 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:50:01.832298 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:01.832317 (+ 19us) server_negotiation.cc:299] Negotiation successful
0619 16:50:01.832347 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 16:50:01.835721 11061 raft_consensus.cc:1261] T 74f16f22ef9c4ef98271b101d33b5636 P c07e63c69fe244b3accacdca2c7489ec [term 1 FOLLOWER]: Refusing update from remote peer 0f9f874691a94ad886e005e4df690c8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:50:01.835724 10927 raft_consensus.cc:1261] T 74f16f22ef9c4ef98271b101d33b5636 P 9c12a35728fe4bfc98fb41abbad60130 [term 1 FOLLOWER]: Refusing update from remote peer 0f9f874691a94ad886e005e4df690c8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:50:01.836369 11132 consensus_queue.cc:1035] T 74f16f22ef9c4ef98271b101d33b5636 P 0f9f874691a94ad886e005e4df690c8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c12a35728fe4bfc98fb41abbad60130" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:50:01.836680 11135 consensus_queue.cc:1035] T 74f16f22ef9c4ef98271b101d33b5636 P 0f9f874691a94ad886e005e4df690c8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07e63c69fe244b3accacdca2c7489ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:50:01.843694 10647 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:50:01.843890 10647 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:50:01.845175 10647 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 90913d6c8bf74afd812111c9a3db25e3: Sending DeleteTablet for 3 replicas of tablet 74f16f22ef9c4ef98271b101d33b5636
I20260619 16:50:01.845631 10773 tablet_service.cc:1514] Processing DeleteTablet for tablet 74f16f22ef9c4ef98271b101d33b5636 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:50:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33392
I20260619 16:50:01.845698 11041 tablet_service.cc:1514] Processing DeleteTablet for tablet 74f16f22ef9c4ef98271b101d33b5636 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:50:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47990
I20260619 16:50:01.845745 10907 tablet_service.cc:1514] Processing DeleteTablet for tablet 74f16f22ef9c4ef98271b101d33b5636 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:50:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57850
I20260619 16:50:01.846144 11147 tablet_replica.cc:330] T 74f16f22ef9c4ef98271b101d33b5636 P 0f9f874691a94ad886e005e4df690c8b: stopping tablet replica
I20260619 16:50:01.846312 11147 raft_consensus.cc:2229] T 74f16f22ef9c4ef98271b101d33b5636 P 0f9f874691a94ad886e005e4df690c8b [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:50:01.846415 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 10710
I20260619 16:50:01.846458 11147 raft_consensus.cc:2258] T 74f16f22ef9c4ef98271b101d33b5636 P 0f9f874691a94ad886e005e4df690c8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:50:01.846025 11149 tablet_replica.cc:330] T 74f16f22ef9c4ef98271b101d33b5636 P 9c12a35728fe4bfc98fb41abbad60130: stopping tablet replica
I20260619 16:50:01.847802 11149 raft_consensus.cc:2229] T 74f16f22ef9c4ef98271b101d33b5636 P 9c12a35728fe4bfc98fb41abbad60130 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:50:01.848009 11149 raft_consensus.cc:2258] T 74f16f22ef9c4ef98271b101d33b5636 P 9c12a35728fe4bfc98fb41abbad60130 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:50:01.848587 11149 ts_tablet_manager.cc:1905] T 74f16f22ef9c4ef98271b101d33b5636 P 9c12a35728fe4bfc98fb41abbad60130: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:50:01.850442 11148 tablet_replica.cc:330] T 74f16f22ef9c4ef98271b101d33b5636 P c07e63c69fe244b3accacdca2c7489ec: stopping tablet replica
I20260619 16:50:01.850705 11148 raft_consensus.cc:2229] T 74f16f22ef9c4ef98271b101d33b5636 P c07e63c69fe244b3accacdca2c7489ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:50:01.851045 11148 raft_consensus.cc:2258] T 74f16f22ef9c4ef98271b101d33b5636 P c07e63c69fe244b3accacdca2c7489ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:50:01.851868 11148 ts_tablet_manager.cc:1905] T 74f16f22ef9c4ef98271b101d33b5636 P c07e63c69fe244b3accacdca2c7489ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:50:01.851963 11149 ts_tablet_manager.cc:1918] T 74f16f22ef9c4ef98271b101d33b5636 P 9c12a35728fe4bfc98fb41abbad60130: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:50:01.852167 11149 log.cc:1196] T 74f16f22ef9c4ef98271b101d33b5636 P 9c12a35728fe4bfc98fb41abbad60130: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-1/wal/wals/74f16f22ef9c4ef98271b101d33b5636
I20260619 16:50:01.852756 11149 ts_tablet_manager.cc:1939] T 74f16f22ef9c4ef98271b101d33b5636 P 9c12a35728fe4bfc98fb41abbad60130: Deleting consensus metadata
I20260619 16:50:01.853833 10633 catalog_manager.cc:4841] TS 9c12a35728fe4bfc98fb41abbad60130 (127.0.98.2:45455): tablet 74f16f22ef9c4ef98271b101d33b5636 (table test-table [id=ec942b46b98e48698b4b3884822a9d81]) successfully deleted
I20260619 16:50:01.855067 11148 ts_tablet_manager.cc:1918] T 74f16f22ef9c4ef98271b101d33b5636 P c07e63c69fe244b3accacdca2c7489ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:50:01.855175 11148 log.cc:1196] T 74f16f22ef9c4ef98271b101d33b5636 P c07e63c69fe244b3accacdca2c7489ec: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781887741903642-392-0/minicluster-data/ts-2/wal/wals/74f16f22ef9c4ef98271b101d33b5636
I20260619 16:50:01.855871 11148 ts_tablet_manager.cc:1939] T 74f16f22ef9c4ef98271b101d33b5636 P c07e63c69fe244b3accacdca2c7489ec: Deleting consensus metadata
W20260619 16:50:01.856199 10634 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35671 (error 108)
I20260619 16:50:01.856673 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 10844
I20260619 16:50:01.857601 10633 catalog_manager.cc:4841] TS c07e63c69fe244b3accacdca2c7489ec (127.0.98.3:45925): tablet 74f16f22ef9c4ef98271b101d33b5636 (table test-table [id=ec942b46b98e48698b4b3884822a9d81]) successfully deleted
I20260619 16:50:01.859598 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:01.856869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35671 (local address 127.0.0.1:33410)
0619 16:50:01.857696 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:01.857766 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":786}
W20260619 16:50:01.859889 10634 catalog_manager.cc:4584] TS 0f9f874691a94ad886e005e4df690c8b (127.0.98.1:35671): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 74f16f22ef9c4ef98271b101d33b5636: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35671: connect: Connection refused (error 111)
I20260619 16:50:01.863952 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 10978
I20260619 16:50:01.869915 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 10616
2026-06-19T16:50:01Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2198 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:50:01 dist-test-slave-jsw9 krb5kdc[11153](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:50:01 dist-test-slave-jsw9 krb5kdc[11153](info): set up 2 sockets
Jun 19 16:50:01 dist-test-slave-jsw9 krb5kdc[11153](info): commencing operation
krb5kdc: starting...
W20260619 16:50:02.923347 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:50:03 dist-test-slave-jsw9 krb5kdc[11153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:50:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:50:03Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260619 16:50:03.151538 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskjYBH_Y/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/postgres -l logfile start
2026-06-19 16:50:06.406 UTC [11183] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 16:50:06.406 UTC [11183] LOG: listening on IPv4 address "127.0.98.20", port 42527
2026-06-19 16:50:06.411 UTC [11183] LOG: listening on Unix socket "/tmp/.s.PGSQL.42527"
2026-06-19 16:50:06.444 UTC [11186] LOG: database system was shut down at 2026-06-19 16:50:04 UTC
2026-06-19 16:50:06.449 UTC [11183] LOG: database system is ready to accept connections
I20260619 16:50:07.331753 392 mini_postgres.cc:96] Postgres bound to 42527
2026-06-19 16:50:07.339 UTC [11194] FATAL: database "slave" does not exist
127.0.98.20:42527 - accepting connections
I20260619 16:50:07.339716 392 mini_ranger.cc:161] Starting Ranger...
I20260619 16:50:07.357270 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 16:50:08.007180 392 mini_ranger.cc:87] Created ranger Postgres database
I20260619 16:50:08.007386 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ranger-admin
2026-06-19 16:50:08,334 [I] DB FLAVOR :POSTGRES
2026-06-19 16:50:08,335 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 16:50:08,335 [I] Checking connection..
2026-06-19 16:50:08,336 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 16:50:08,642 [I] Checking connection passed.
2026-06-19 16:50:08,642 [I] --------- Verifying version history table ---------
2026-06-19 16:50:08,642 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:50:08,965 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 16:50:08,966 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:50:09,323 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 16:50:09,323 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 16:50:09,323 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 16:50:09.632 UTC [11258] WARNING: there is no transaction in progress
2026-06-19 16:50:09,640 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 16:50:09,640 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:50:09,930 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 16:50:09,930 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 16:50:09,931 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 16:50:10,256 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 16:50:10,579 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-jsw9.c.gcp-upstream.internal','N') ;"
2026-06-19 16:50:10,861 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 16:50:11,212 [I] Table x_portal_user does not exist in database ranger
2026-06-19 16:50:11,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 16:50:11,521 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 16:50:11,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 16:50:11,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 16:50:12,163 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 16:50:12,164 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 16:50:13.439 UTC [11366] WARNING: there is no transaction in progress
2026-06-19 16:50:13.459 UTC [11366] WARNING: there is no transaction in progress
2026-06-19 16:50:13.480 UTC [11366] WARNING: there is no transaction in progress
2026-06-19 16:50:13.499 UTC [11366] WARNING: there is no transaction in progress
2026-06-19 16:50:13.522 UTC [11366] WARNING: there is no transaction in progress
2026-06-19 16:50:13.550 UTC [11366] WARNING: there is no transaction in progress
2026-06-19 16:50:13.627 UTC [11366] WARNING: there is no transaction in progress
2026-06-19 16:50:13.636 UTC [11366] WARNING: there is no transaction in progress
2026-06-19 16:50:13.656 UTC [11366] WARNING: there is no transaction in progress
2026-06-19 16:50:13.669 UTC [11366] WARNING: there is no transaction in progress
2026-06-19 16:50:13.681 UTC [11366] WARNING: there is no transaction in progress
2026-06-19 16:50:13.700 UTC [11366] WARNING: there is no transaction in progress
2026-06-19 16:50:13.712 UTC [11366] WARNING: there is no transaction in progress
2026-06-19 16:50:13.720 UTC [11366] WARNING: there is no transaction in progress
2026-06-19 16:50:13.730 UTC [11366] WARNING: there is no transaction in progress
2026-06-19 16:50:13.740 UTC [11366] WARNING: there is no transaction in progress
2026-06-19 16:50:14,640 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 16:50:14,641 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 16:50:14,929 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 16:50:14,929 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 16:50:15,259 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 16:50:15,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 16:50:15,566 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 16:50:15,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 16:50:15,863 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 16:50:16,193 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-jsw9.c.gcp-upstream.internal';"
2026-06-19 16:50:16,502 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 16:50:16,502 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42527/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 16:50:16,784 [I] DB_PATCHES have already been applied
I20260619 16:50:16.791579 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskjYBH_Y/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 16:50:16.791688 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260619 16:50:16.795326 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:41513
Jun 19, 2026 4:50:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 4:50:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 4:50:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 4:50:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 4:50:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 4:50:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 4:50:17 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 16:50:18.909328 392 mini_ranger.cc:160] Time spent starting Ranger: real 11.570s user 0.003s sys 0.010s
Jun 19, 2026 4:50:41 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 4:50:41 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 19, 2026 4:50:41 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 4:50:41 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 4:50:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 4:50:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 4:50:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 4:50:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 4:50:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 4:50:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 4:50:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 4:50:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 4:50:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 4:50:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 4:50:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 4:50:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 4:50:41 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 4:50:42 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 16:50:43.286547 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 16:50:43 dist-test-slave-jsw9 krb5kdc[11153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887843, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 16:50:43.376830 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 16:50:43.389796 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 16:50:44.473938 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 16:50:44.474030 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ranger-kms
I20260619 16:50:44.474047 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:42527
2026-06-19 16:50:44,706 [I] DB FLAVOR :POSTGRES
2026-06-19 16:50:44,706 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 16:50:44,706 [I] Checking connection
2026-06-19 16:50:44,707 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42527/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 16:50:45,028 [I] connection success
2026-06-19 16:50:45,028 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 16:50:45,028 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 16:50:45,028 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42527/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 16:50:45,402 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 16:50:45,402 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 16:50:45,403 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 16:50:45,403 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42527/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 16:50:45,731 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 16:50:46.378379 392 mini_ranger_kms.cc:310] Created kms service
I20260619 16:50:46.632375 392 mini_ranger_kms.cc:326] Created kudu user
I20260619 16:50:46.688585 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 16:50:46 dist-test-slave-jsw9 krb5kdc[11153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887843, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:50:47.195668 392 mini_ranger_kms.cc:384] Added ranger policy
I20260619 16:50:47.195819 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 16:50:47.195840 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260619 16:50:47.201530 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11548
I20260619 16:50:47.201685 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:53269
Jun 19, 2026 4:50:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 4:50:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 4:50:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 4:50:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 4:50:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 4:50:49 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 16:50:49.351238 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.974s user 0.006s sys 0.206s
I20260619 16:50:49.351358 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 16:50:49.351449 392 mini_ranger_kms.cc:401] 127.0.98.20:53269/kms/v1/keys
Jun 19 16:50:49 dist-test-slave-jsw9 krb5kdc[11153](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781887849, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 16:50:50 dist-test-slave-jsw9 krb5kdc[11153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887849, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 16:50:50 dist-test-slave-jsw9 krb5kdc[11153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887849, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 16:50:53 dist-test-slave-jsw9 krb5kdc[11153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:50:53.619431 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40299
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40299
--ranger_config_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53269/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:53.714412 11598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:53.714675 11598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:53.714712 11598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:53.716197 11598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:50:53.716286 11598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:53.716310 11598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:53.716327 11598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:50:53.716342 11598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:50:53.718693 11598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:53269/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40299
--ranger_config_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:50:53.719014 11598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:53.719282 11598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:53.725941 11603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:53.726078 11598 server_base.cc:1034] running on GCE node
W20260619 16:50:53.726207 11606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:53.726034 11604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:53.726585 11598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:53.727000 11598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:53.728245 11598 hybrid_clock.cc:648] HybridClock initialized: now 1781887853728165 us; error 34 us; skew 500 ppm
Jun 19 16:50:53 dist-test-slave-jsw9 krb5kdc[11153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887853, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:53.735872 11598 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:50:53.736589 11598 webserver.cc:460] Webserver started at http://127.0.98.62:41583/ using document root <none> and password file <none>
I20260619 16:50:53.736881 11598 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:53.736941 11598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:53.737102 11598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:50:53 dist-test-slave-jsw9 krb5kdc[11153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887853, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:50:53.797528 11598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "65fadfeef38642428dc241e1852c7f24"
format_stamp: "Formatted at 2026-06-19 16:50:53 on dist-test-slave-jsw9"
server_key: "78585427f80cb64197b9d59929afd264"
server_key_iv: "3a2e7bdddab85705f0105552a707904b"
server_key_version: "kuduclusterkey@0"
I20260619 16:50:53.798027 11598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "65fadfeef38642428dc241e1852c7f24"
format_stamp: "Formatted at 2026-06-19 16:50:53 on dist-test-slave-jsw9"
server_key: "78585427f80cb64197b9d59929afd264"
server_key_iv: "3a2e7bdddab85705f0105552a707904b"
server_key_version: "kuduclusterkey@0"
I20260619 16:50:53.799708 11598 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:50:53.815281 11613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:53.815508 11598 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:50:53.815603 11598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "65fadfeef38642428dc241e1852c7f24"
format_stamp: "Formatted at 2026-06-19 16:50:53 on dist-test-slave-jsw9"
server_key: "78585427f80cb64197b9d59929afd264"
server_key_iv: "3a2e7bdddab85705f0105552a707904b"
server_key_version: "kuduclusterkey@0"
I20260619 16:50:53.815733 11598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:53.833529 11598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:53.835829 11598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:53.836000 11598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:53.842130 11598 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40299
I20260619 16:50:53.842146 11665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40299 every 8 connection(s)
I20260619 16:50:53.842635 11598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:50:53.843509 11666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:53.847082 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 11598
I20260619 16:50:53.847193 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:50:53.847316 11666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65fadfeef38642428dc241e1852c7f24: Bootstrap starting.
I20260619 16:50:53.847385 392 external_mini_cluster.cc:1428] Setting key 52727e0dd2269c6bbd93ffb30385f84e
I20260619 16:50:53.848181 11666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65fadfeef38642428dc241e1852c7f24: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:53.848526 11666 log.cc:824] T 00000000000000000000000000000000 P 65fadfeef38642428dc241e1852c7f24: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:53.849480 11666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65fadfeef38642428dc241e1852c7f24: No bootstrap required, opened a new log
I20260619 16:50:53.852545 11666 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65fadfeef38642428dc241e1852c7f24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65fadfeef38642428dc241e1852c7f24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40299 } }
I20260619 16:50:53.852890 11666 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65fadfeef38642428dc241e1852c7f24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:53.852947 11666 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65fadfeef38642428dc241e1852c7f24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65fadfeef38642428dc241e1852c7f24, State: Initialized, Role: FOLLOWER
I20260619 16:50:53.853269 11666 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65fadfeef38642428dc241e1852c7f24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65fadfeef38642428dc241e1852c7f24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40299 } }
I20260619 16:50:53.853382 11666 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65fadfeef38642428dc241e1852c7f24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:50:53.853446 11666 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65fadfeef38642428dc241e1852c7f24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:50:53.853503 11666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65fadfeef38642428dc241e1852c7f24 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:53.854430 11666 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65fadfeef38642428dc241e1852c7f24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65fadfeef38642428dc241e1852c7f24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40299 } }
I20260619 16:50:53.854594 11666 leader_election.cc:302] T 00000000000000000000000000000000 P 65fadfeef38642428dc241e1852c7f24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65fadfeef38642428dc241e1852c7f24; no voters:
I20260619 16:50:53.854794 11666 leader_election.cc:288] T 00000000000000000000000000000000 P 65fadfeef38642428dc241e1852c7f24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:50:53.854831 11671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65fadfeef38642428dc241e1852c7f24 [term 1 FOLLOWER]: Leader election won for term 1
Jun 19 16:50:53 dist-test-slave-jsw9 krb5kdc[11153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:50:53.855137 11671 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65fadfeef38642428dc241e1852c7f24 [term 1 LEADER]: Becoming Leader. State: Replica: 65fadfeef38642428dc241e1852c7f24, State: Running, Role: LEADER
I20260619 16:50:53.855221 11666 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65fadfeef38642428dc241e1852c7f24 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:50:53.855641 11671 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65fadfeef38642428dc241e1852c7f24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65fadfeef38642428dc241e1852c7f24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40299 } }
I20260619 16:50:53.856163 11671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65fadfeef38642428dc241e1852c7f24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65fadfeef38642428dc241e1852c7f24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65fadfeef38642428dc241e1852c7f24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40299 } } }
I20260619 16:50:53.856321 11671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65fadfeef38642428dc241e1852c7f24 [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:53.856516 11671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65fadfeef38642428dc241e1852c7f24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65fadfeef38642428dc241e1852c7f24. Latest consensus state: current_term: 1 leader_uuid: "65fadfeef38642428dc241e1852c7f24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65fadfeef38642428dc241e1852c7f24" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40299 } } }
I20260619 16:50:53.856626 11671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65fadfeef38642428dc241e1852c7f24 [sys.catalog]: This master's current role is: LEADER
I20260619 16:50:53.856787 11666 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 16:50:53.861069 11684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:50:53.862005 11684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:50:53.865727 11669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:53.848011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59142 (local address 127.0.98.62:40299)
0619 16:50:53.848250 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:53.848262 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:53.848298 (+ 36us) server_negotiation.cc:407] Connection header received
0619 16:50:53.848985 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:53.848998 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:53.849075 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:53.849245 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:53.850024 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:53.851414 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:53.852218 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:53.852697 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:53.852800 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:53.855624 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:53.855629 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:53.855630 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:53.855644 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:53.860176 (+ 4532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:53.860756 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:53.860761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:53.860761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:53.860977 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:53.862475 (+ 1498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:53.862476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:53.862476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:53.863228 (+ 752us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:53.863524 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:53.864144 (+ 620us) server_negotiation.cc:299] Negotiation successful
0619 16:50:53.865341 (+ 1197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":96,"threads_started":1}
I20260619 16:50:53.870899 11684 catalog_manager.cc:1347] Generated new cluster ID: 7e525b28b98b47a8bb7826211d6cc252
I20260619 16:50:53.870981 11684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:50:53.885253 11684 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:50:53.887039 11684 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:50:53.892663 11684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65fadfeef38642428dc241e1852c7f24: Generated new TSK 0
I20260619 16:50:53.893076 11684 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:50:54.806751 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53269/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40299
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:54.949486 11711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:54.949875 11711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:54.949926 11711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:54.952435 11711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:54.952557 11711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:54.952617 11711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:50:54.955622 11711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:53269/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:50:54.956020 11711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:54.956519 11711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:54.963312 11716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:54.963557 11711 server_base.cc:1034] running on GCE node
W20260619 16:50:54.963331 11717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:54.963706 11719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:54.964073 11711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:54.964398 11711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:54.965574 11711 hybrid_clock.cc:648] HybridClock initialized: now 1781887854965550 us; error 49 us; skew 500 ppm
Jun 19 16:50:54 dist-test-slave-jsw9 krb5kdc[11153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887854, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:54.969429 11711 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:50:54.969991 11711 webserver.cc:460] Webserver started at http://127.0.98.1:44949/ using document root <none> and password file <none>
I20260619 16:50:54.970268 11711 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:54.970323 11711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:54.970460 11711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:50:54 dist-test-slave-jsw9 krb5kdc[11153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887854, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:50:54.988627 11711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-0/data/instance:
uuid: "a61795d2849947ef97be8e4b1709dddf"
format_stamp: "Formatted at 2026-06-19 16:50:54 on dist-test-slave-jsw9"
server_key: "5113ab86a446f4b17d3a576311f94735"
server_key_iv: "0a168d2d2c926935dd2d0521a052cd1f"
server_key_version: "kuduclusterkey@0"
I20260619 16:50:54.989282 11711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a61795d2849947ef97be8e4b1709dddf"
format_stamp: "Formatted at 2026-06-19 16:50:54 on dist-test-slave-jsw9"
server_key: "5113ab86a446f4b17d3a576311f94735"
server_key_iv: "0a168d2d2c926935dd2d0521a052cd1f"
server_key_version: "kuduclusterkey@0"
I20260619 16:50:54.991462 11711 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:50:55.009987 11727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:55.010324 11711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 16:50:55.010444 11711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-0/wal
uuid: "a61795d2849947ef97be8e4b1709dddf"
format_stamp: "Formatted at 2026-06-19 16:50:54 on dist-test-slave-jsw9"
server_key: "5113ab86a446f4b17d3a576311f94735"
server_key_iv: "0a168d2d2c926935dd2d0521a052cd1f"
server_key_version: "kuduclusterkey@0"
I20260619 16:50:55.010598 11711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:55.030968 11711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:55.033838 11711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:55.034035 11711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:55.034380 11711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:50:55.034816 11711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:50:55.034868 11711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:55.034907 11711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:50:55.034929 11711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:55.044358 11711 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44263
I20260619 16:50:55.044420 11841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44263 every 8 connection(s)
I20260619 16:50:55.044955 11711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:50:55.049013 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 11711
I20260619 16:50:55.049135 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:50:55.049337 392 external_mini_cluster.cc:1428] Setting key 7b3981ac8e6cde9b57107d493bd36d1f
Jun 19 16:50:55 dist-test-slave-jsw9 krb5kdc[11153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887854, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:50:55.058339 11844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.046286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51321 (local address 127.0.98.62:40299)
0619 16:50:55.046816 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:55.046819 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:55.047111 (+ 292us) server_negotiation.cc:407] Connection header received
0619 16:50:55.047868 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:55.047870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:55.047885 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:55.048003 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:55.048855 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.049687 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.051170 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.051484 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.051566 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:55.053936 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:55.053938 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:55.053938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:55.053948 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:55.055743 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:55.056972 (+ 1229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:55.056974 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:55.056974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:55.057065 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:55.057395 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:55.057396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:55.057397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:55.057561 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:55.057609 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:55.058066 (+ 457us) server_negotiation.cc:299] Negotiation successful
0619 16:50:55.058181 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":71,"threads_started":1}
I20260619 16:50:55.059377 11845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.046555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40299 (local address 127.0.98.1:51321)
0619 16:50:55.046890 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:55.046946 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:55.047724 (+ 778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:55.048104 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:55.048109 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:55.048401 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:55.048731 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:55.048737 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.049794 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.049795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:55.051070 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:55.051073 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.051579 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.051580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:55.051743 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:55.052385 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:55.052395 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:55.053832 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:55.055846 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:55.055856 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:55.055859 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:55.056394 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:55.057146 (+ 752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:55.057147 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:55.057148 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:55.057316 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:55.057654 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:55.057661 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:55.057886 (+ 225us) client_negotiation.cc:769] Sending connection context
0619 16:50:55.058915 (+ 1029us) client_negotiation.cc:240] Negotiation successful
0619 16:50:55.059117 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 16:50:55.059984 11842 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40299
I20260619 16:50:55.060108 11842 heartbeater.cc:464] Registering TS with master...
I20260619 16:50:55.060345 11842 heartbeater.cc:511] Master 127.0.98.62:40299 requested a full tablet report, sending...
I20260619 16:50:55.061007 11630 ts_manager.cc:194] Registered new tserver with Master: a61795d2849947ef97be8e4b1709dddf (127.0.98.1:44263)
I20260619 16:50:55.062043 11630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51321
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:50:55.082239 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53269/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40299
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:50:55 dist-test-slave-jsw9 krb5kdc[11153](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781887855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 16:50:55.223418 11848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:55.223774 11848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:55.223822 11848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:55.226275 11848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:55.226430 11848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:55.226517 11848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:50:55.229598 11848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:53269/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:50:55.230123 11848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:55.230538 11848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:55.238978 11856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:55.239476 11858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:55.239924 11855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:55.241268 11848 server_base.cc:1034] running on GCE node
I20260619 16:50:55.241551 11848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:55.242048 11848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:55.243340 11848 hybrid_clock.cc:648] HybridClock initialized: now 1781887855243310 us; error 52 us; skew 500 ppm
Jun 19 16:50:55 dist-test-slave-jsw9 krb5kdc[11153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:55.250180 11848 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:50:55.250830 11848 webserver.cc:460] Webserver started at http://127.0.98.2:45279/ using document root <none> and password file <none>
I20260619 16:50:55.251178 11848 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:55.251238 11848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:55.251408 11848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:50:55 dist-test-slave-jsw9 krb5kdc[11153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:50:55.289023 11848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-1/data/instance:
uuid: "7cc58d26c1154b37bd7f43bdc47d0449"
format_stamp: "Formatted at 2026-06-19 16:50:55 on dist-test-slave-jsw9"
server_key: "524722f1e614ea241a18f9cebbfd5f8c"
server_key_iv: "1a6f7822b5b94e0e67ea44fe270311ae"
server_key_version: "kuduclusterkey@0"
I20260619 16:50:55.289775 11848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7cc58d26c1154b37bd7f43bdc47d0449"
format_stamp: "Formatted at 2026-06-19 16:50:55 on dist-test-slave-jsw9"
server_key: "524722f1e614ea241a18f9cebbfd5f8c"
server_key_iv: "1a6f7822b5b94e0e67ea44fe270311ae"
server_key_version: "kuduclusterkey@0"
I20260619 16:50:55.295398 11848 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.000s
I20260619 16:50:55.322935 11866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:55.323176 11848 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:50:55.323282 11848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-1/wal
uuid: "7cc58d26c1154b37bd7f43bdc47d0449"
format_stamp: "Formatted at 2026-06-19 16:50:55 on dist-test-slave-jsw9"
server_key: "524722f1e614ea241a18f9cebbfd5f8c"
server_key_iv: "1a6f7822b5b94e0e67ea44fe270311ae"
server_key_version: "kuduclusterkey@0"
I20260619 16:50:55.323422 11848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:55.359395 11848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:55.362459 11848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:55.362674 11848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:55.362955 11848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:50:55.363399 11848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:50:55.363452 11848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:55.363490 11848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:50:55.363515 11848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:55.373172 11848 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39233
I20260619 16:50:55.373721 11848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:50:55.378754 11979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39233 every 8 connection(s)
I20260619 16:50:55.382148 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 11848
I20260619 16:50:55.382299 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:50:55.382531 392 external_mini_cluster.cc:1428] Setting key 786d08dbcc3ec00e3032d3e491d775a6
Jun 19 16:50:55 dist-test-slave-jsw9 krb5kdc[11153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:50:55.392689 11844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.380132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35321 (local address 127.0.98.62:40299)
0619 16:50:55.380214 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:55.380217 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:55.380572 (+ 355us) server_negotiation.cc:407] Connection header received
0619 16:50:55.381431 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:55.381432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:55.381445 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:55.381476 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:55.382518 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.383378 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.385207 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.385524 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.385577 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:55.388738 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:55.388740 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:55.388740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:55.388749 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:55.390634 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:55.391283 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:55.391285 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:55.391285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:55.391377 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:55.391716 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:55.391717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:55.391717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:55.391883 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:55.391931 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:55.392435 (+ 504us) server_negotiation.cc:299] Negotiation successful
0619 16:50:55.392568 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 16:50:55.395133 11982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.379745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40299 (local address 127.0.98.2:35321)
0619 16:50:55.380411 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:55.380468 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:55.381280 (+ 812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:55.381694 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:55.381699 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:55.382017 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:55.382396 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:55.382404 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.383512 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.383513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:55.385091 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:55.385096 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.385786 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.385788 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:55.385964 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:55.386723 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:55.386737 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:55.388595 (+ 1858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:55.390745 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:55.390757 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:55.390761 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:55.391133 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:55.391459 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:55.391460 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:55.391460 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:55.391644 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:55.391991 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:55.391998 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:55.392230 (+ 232us) client_negotiation.cc:769] Sending connection context
0619 16:50:55.393283 (+ 1053us) client_negotiation.cc:240] Negotiation successful
0619 16:50:55.394606 (+ 1323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":137,"threads_started":1}
I20260619 16:50:55.397657 11980 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40299
I20260619 16:50:55.397802 11980 heartbeater.cc:464] Registering TS with master...
I20260619 16:50:55.398090 11980 heartbeater.cc:511] Master 127.0.98.62:40299 requested a full tablet report, sending...
I20260619 16:50:55.398691 11630 ts_manager.cc:194] Registered new tserver with Master: 7cc58d26c1154b37bd7f43bdc47d0449 (127.0.98.2:39233)
I20260619 16:50:55.399317 11630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35321
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:50:55.437140 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:53269/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40299
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:50:55.584872 11985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:50:55.585249 11985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:50:55.585296 11985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:50:55.587447 11985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:50:55.587540 11985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:50:55.587594 11985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:50:55.590232 11985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:53269/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:50:55.590590 11985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:50:55.590935 11985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:50:55.596520 11990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:55.596798 11991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:50:55.596978 11993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:50:55.597213 11985 server_base.cc:1034] running on GCE node
I20260619 16:50:55.597434 11985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:50:55.597710 11985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:50:55.598876 11985 hybrid_clock.cc:648] HybridClock initialized: now 1781887855598856 us; error 34 us; skew 500 ppm
Jun 19 16:50:55 dist-test-slave-jsw9 krb5kdc[11153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:50:55.601722 11985 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:50:55.602228 11985 webserver.cc:460] Webserver started at http://127.0.98.3:38995/ using document root <none> and password file <none>
I20260619 16:50:55.602483 11985 fs_manager.cc:359] Metadata directory not provided
I20260619 16:50:55.602552 11985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:50:55.602710 11985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:50:55 dist-test-slave-jsw9 krb5kdc[11153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:50:55.616542 11985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-2/data/instance:
uuid: "e144352b42cf4e9aab14a5b74482a3c5"
format_stamp: "Formatted at 2026-06-19 16:50:55 on dist-test-slave-jsw9"
server_key: "4b6c4119d165b2bfce43a5c8907ec4a6"
server_key_iv: "c73585ada26a596ce2f8980390710786"
server_key_version: "kuduclusterkey@0"
I20260619 16:50:55.616990 11985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e144352b42cf4e9aab14a5b74482a3c5"
format_stamp: "Formatted at 2026-06-19 16:50:55 on dist-test-slave-jsw9"
server_key: "4b6c4119d165b2bfce43a5c8907ec4a6"
server_key_iv: "c73585ada26a596ce2f8980390710786"
server_key_version: "kuduclusterkey@0"
I20260619 16:50:55.618557 11985 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 16:50:55.634337 12000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:55.634583 11985 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:50:55.634690 11985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-2/wal
uuid: "e144352b42cf4e9aab14a5b74482a3c5"
format_stamp: "Formatted at 2026-06-19 16:50:55 on dist-test-slave-jsw9"
server_key: "4b6c4119d165b2bfce43a5c8907ec4a6"
server_key_iv: "c73585ada26a596ce2f8980390710786"
server_key_version: "kuduclusterkey@0"
I20260619 16:50:55.634800 11985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:50:55.660851 11985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:50:55.664426 11985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:50:55.664650 11985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:50:55.664960 11985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:50:55.665465 11985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:50:55.665521 11985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:55.665562 11985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:50:55.665580 11985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:50:55.672938 11985 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46031
I20260619 16:50:55.673476 11985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:50:55.676096 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 11985
I20260619 16:50:55.676211 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:50:55.676433 392 external_mini_cluster.cc:1428] Setting key 61466b33fb4f9895e4698fe2ba54ee8c
I20260619 16:50:55.686854 12113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46031 every 8 connection(s)
Jun 19 16:50:55 dist-test-slave-jsw9 krb5kdc[11153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:50:55.701155 11844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.687346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57921 (local address 127.0.98.62:40299)
0619 16:50:55.687404 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:55.687407 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:55.687902 (+ 495us) server_negotiation.cc:407] Connection header received
0619 16:50:55.688805 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:55.688807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:55.688821 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:55.688855 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:55.689868 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.690735 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.692767 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.693071 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.693125 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:55.697332 (+ 4207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:55.697335 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:55.697335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:55.697349 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:55.699199 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:55.699837 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:55.699839 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:55.699839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:55.699908 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:55.700228 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:55.700230 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:55.700230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:55.700396 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:55.700443 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:55.700928 (+ 485us) server_negotiation.cc:299] Negotiation successful
0619 16:50:55.701037 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 16:50:55.702086 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.687355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40299 (local address 127.0.98.3:57921)
0619 16:50:55.687606 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:55.687808 (+ 202us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:55.688665 (+ 857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:55.689073 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:55.689078 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:55.689391 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:50:55.689738 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:55.689744 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.690867 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.690868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:55.692654 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:55.692658 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.693241 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.693242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:55.693414 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:55.695132 (+ 1718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:50:55.695145 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:50:55.697175 (+ 2030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:50:55.699314 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:55.699324 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:55.699328 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:55.699738 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:55.699979 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:50:55.699980 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:50:55.699980 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:50:55.700150 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:50:55.700486 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:50:55.700495 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:50:55.700753 (+ 258us) client_negotiation.cc:769] Sending connection context
0619 16:50:55.701565 (+ 812us) client_negotiation.cc:240] Negotiation successful
0619 16:50:55.701800 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":151,"threads_started":1}
I20260619 16:50:55.702986 12114 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40299
I20260619 16:50:55.703120 12114 heartbeater.cc:464] Registering TS with master...
I20260619 16:50:55.703347 12114 heartbeater.cc:511] Master 127.0.98.62:40299 requested a full tablet report, sending...
I20260619 16:50:55.703783 11630 ts_manager.cc:194] Registered new tserver with Master: e144352b42cf4e9aab14a5b74482a3c5 (127.0.98.3:46031)
I20260619 16:50:55.704361 11630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57921
I20260619 16:50:55.705101 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:50:55.715610 11844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.707035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59156 (local address 127.0.98.62:40299)
0619 16:50:55.707096 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:55.707098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:55.707281 (+ 183us) server_negotiation.cc:407] Connection header received
0619 16:50:55.707381 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:55.707383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:55.707394 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:55.707425 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:50:55.708338 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.709167 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.710333 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.710675 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.710722 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:55.712477 (+ 1755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:50:55.712479 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:50:55.712480 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:50:55.712488 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:50:55.714266 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:55.714688 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:55.714689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:55.714689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:55.714754 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:50:55.714944 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:50:55.714945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:50:55.714946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:50:55.715092 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:50:55.715180 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:55.715388 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 16:50:55.715481 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 16:50:55.718369 11630 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:50:55.721696 11630 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:50:55.748805 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.737853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39233 (local address 127.0.0.1:39816)
0619 16:50:55.742758 (+ 4905us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:55.742772 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:55.742844 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:55.743871 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:55.743876 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:55.743879 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:55.743997 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:55.743999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.745431 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.745432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:55.747261 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:55.747264 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.748443 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.748444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:55.748538 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:55.748555 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 16:50:55.748629 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 16:50:55.748694 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4876,"mutex_wait_us":324,"thread_start_us":69,"threads_started":1}
I20260619 16:50:55.749956 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.738018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39816 (local address 127.0.98.2:39233)
0619 16:50:55.743537 (+ 5519us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:55.743541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:55.743553 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:50:55.743602 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:55.743605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:55.743626 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:55.743715 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:55.744096 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.745311 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.747402 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.748226 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.748297 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:55.748404 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:55.749780 (+ 1376us) server_negotiation.cc:299] Negotiation successful
0619 16:50:55.749839 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5469,"thread_start_us":102,"threads_started":1}
I20260619 16:50:55.751009 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.738812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44263 (local address 127.0.0.1:51402)
0619 16:50:55.746291 (+ 7479us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:55.746305 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:55.746379 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:55.746733 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:55.746734 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:55.746736 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:55.746793 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:55.746795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.748093 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.748094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:55.749170 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:55.749173 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.750759 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.750760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:55.750829 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:55.750834 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:50:55.750881 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 16:50:55.750932 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7455,"thread_start_us":86,"threads_started":1}
I20260619 16:50:55.750936 11914 tablet_service.cc:1467] Processing CreateTablet for tablet 6d956c8aaee84bfe98f7e86ba9e4cbac (DEFAULT_TABLE table=test-table [id=6d14ba22849a4428b0ef26bbe3ac356f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:50:55.751315 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.738417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51402 (local address 127.0.98.1:44263)
0619 16:50:55.743407 (+ 4990us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:55.743415 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:55.746456 (+ 3041us) server_negotiation.cc:407] Connection header received
0619 16:50:55.746508 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:55.746511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:55.746544 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:55.746630 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:55.746884 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.747988 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.749279 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.750140 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.750251 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:55.750323 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:55.751180 (+ 857us) server_negotiation.cc:299] Negotiation successful
0619 16:50:55.751239 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4401,"thread_start_us":69,"threads_started":1}
I20260619 16:50:55.751468 11914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d956c8aaee84bfe98f7e86ba9e4cbac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:55.752010 11776 tablet_service.cc:1467] Processing CreateTablet for tablet 6d956c8aaee84bfe98f7e86ba9e4cbac (DEFAULT_TABLE table=test-table [id=6d14ba22849a4428b0ef26bbe3ac356f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:50:55.752362 11776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d956c8aaee84bfe98f7e86ba9e4cbac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:55.756433 12131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.737853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46031 (local address 127.0.0.1:43304)
0619 16:50:55.742443 (+ 4590us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:55.742535 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:55.742665 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:55.750678 (+ 8013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:55.750679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:55.750683 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:55.750736 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:55.750738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.752520 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.752522 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:55.754705 (+ 2183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:55.754710 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.756040 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.756041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:55.756121 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:55.756128 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 16:50:55.756158 (+ 30us) client_negotiation.cc:240] Negotiation successful
0619 16:50:55.756213 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4526,"thread_start_us":165,"threads_started":1}
I20260619 16:50:55.757256 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:55.738399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43304 (local address 127.0.98.3:46031)
0619 16:50:55.750456 (+ 12057us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:55.750462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:55.750474 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:50:55.750521 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:55.750523 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:55.750554 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:55.750648 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:55.750835 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.752405 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.754865 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:55.755926 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:55.756646 (+ 720us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:55.756764 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:55.756796 (+ 32us) server_negotiation.cc:299] Negotiation successful
0619 16:50:55.756848 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11998,"thread_start_us":87,"threads_started":1}
I20260619 16:50:55.758307 12037 tablet_service.cc:1467] Processing CreateTablet for tablet 6d956c8aaee84bfe98f7e86ba9e4cbac (DEFAULT_TABLE table=test-table [id=6d14ba22849a4428b0ef26bbe3ac356f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:50:55.758771 12037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d956c8aaee84bfe98f7e86ba9e4cbac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:50:55.762410 12133 tablet_bootstrap.cc:492] T 6d956c8aaee84bfe98f7e86ba9e4cbac P a61795d2849947ef97be8e4b1709dddf: Bootstrap starting.
I20260619 16:50:55.763470 12133 tablet_bootstrap.cc:654] T 6d956c8aaee84bfe98f7e86ba9e4cbac P a61795d2849947ef97be8e4b1709dddf: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:55.763955 12133 log.cc:824] T 6d956c8aaee84bfe98f7e86ba9e4cbac P a61795d2849947ef97be8e4b1709dddf: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:55.764369 12134 tablet_bootstrap.cc:492] T 6d956c8aaee84bfe98f7e86ba9e4cbac P e144352b42cf4e9aab14a5b74482a3c5: Bootstrap starting.
I20260619 16:50:55.765501 12134 tablet_bootstrap.cc:654] T 6d956c8aaee84bfe98f7e86ba9e4cbac P e144352b42cf4e9aab14a5b74482a3c5: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:55.765851 12134 log.cc:824] T 6d956c8aaee84bfe98f7e86ba9e4cbac P e144352b42cf4e9aab14a5b74482a3c5: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:55.765842 12132 tablet_bootstrap.cc:492] T 6d956c8aaee84bfe98f7e86ba9e4cbac P 7cc58d26c1154b37bd7f43bdc47d0449: Bootstrap starting.
I20260619 16:50:55.766791 12134 tablet_bootstrap.cc:492] T 6d956c8aaee84bfe98f7e86ba9e4cbac P e144352b42cf4e9aab14a5b74482a3c5: No bootstrap required, opened a new log
I20260619 16:50:55.766871 12134 ts_tablet_manager.cc:1397] T 6d956c8aaee84bfe98f7e86ba9e4cbac P e144352b42cf4e9aab14a5b74482a3c5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:50:55.769966 12134 raft_consensus.cc:348] T 6d956c8aaee84bfe98f7e86ba9e4cbac P e144352b42cf4e9aab14a5b74482a3c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e144352b42cf4e9aab14a5b74482a3c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46031 } } peers { permanent_uuid: "7cc58d26c1154b37bd7f43bdc47d0449" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39233 } } peers { permanent_uuid: "a61795d2849947ef97be8e4b1709dddf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44263 } }
I20260619 16:50:55.770180 12134 raft_consensus.cc:374] T 6d956c8aaee84bfe98f7e86ba9e4cbac P e144352b42cf4e9aab14a5b74482a3c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:55.770248 12134 raft_consensus.cc:729] T 6d956c8aaee84bfe98f7e86ba9e4cbac P e144352b42cf4e9aab14a5b74482a3c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e144352b42cf4e9aab14a5b74482a3c5, State: Initialized, Role: FOLLOWER
I20260619 16:50:55.770766 12132 tablet_bootstrap.cc:654] T 6d956c8aaee84bfe98f7e86ba9e4cbac P 7cc58d26c1154b37bd7f43bdc47d0449: Neither blocks nor log segments found. Creating new log.
I20260619 16:50:55.770946 12134 consensus_queue.cc:260] T 6d956c8aaee84bfe98f7e86ba9e4cbac P e144352b42cf4e9aab14a5b74482a3c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e144352b42cf4e9aab14a5b74482a3c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46031 } } peers { permanent_uuid: "7cc58d26c1154b37bd7f43bdc47d0449" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39233 } } peers { permanent_uuid: "a61795d2849947ef97be8e4b1709dddf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44263 } }
I20260619 16:50:55.771282 12134 ts_tablet_manager.cc:1428] T 6d956c8aaee84bfe98f7e86ba9e4cbac P e144352b42cf4e9aab14a5b74482a3c5: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 16:50:55.771435 12114 heartbeater.cc:503] Master 127.0.98.62:40299 was elected leader, sending a full tablet report...
I20260619 16:50:55.771437 12132 log.cc:824] T 6d956c8aaee84bfe98f7e86ba9e4cbac P 7cc58d26c1154b37bd7f43bdc47d0449: Log is configured to *not* fsync() on all Append() calls
I20260619 16:50:55.779232 12133 tablet_bootstrap.cc:492] T 6d956c8aaee84bfe98f7e86ba9e4cbac P a61795d2849947ef97be8e4b1709dddf: No bootstrap required, opened a new log
I20260619 16:50:55.779496 12133 ts_tablet_manager.cc:1397] T 6d956c8aaee84bfe98f7e86ba9e4cbac P a61795d2849947ef97be8e4b1709dddf: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20260619 16:50:55.782864 12132 tablet_bootstrap.cc:492] T 6d956c8aaee84bfe98f7e86ba9e4cbac P 7cc58d26c1154b37bd7f43bdc47d0449: No bootstrap required, opened a new log
I20260619 16:50:55.783098 12132 ts_tablet_manager.cc:1397] T 6d956c8aaee84bfe98f7e86ba9e4cbac P 7cc58d26c1154b37bd7f43bdc47d0449: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20260619 16:50:55.785727 12132 raft_consensus.cc:348] T 6d956c8aaee84bfe98f7e86ba9e4cbac P 7cc58d26c1154b37bd7f43bdc47d0449 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e144352b42cf4e9aab14a5b74482a3c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46031 } } peers { permanent_uuid: "7cc58d26c1154b37bd7f43bdc47d0449" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39233 } } peers { permanent_uuid: "a61795d2849947ef97be8e4b1709dddf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44263 } }
I20260619 16:50:55.786816 12133 raft_consensus.cc:348] T 6d956c8aaee84bfe98f7e86ba9e4cbac P a61795d2849947ef97be8e4b1709dddf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e144352b42cf4e9aab14a5b74482a3c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46031 } } peers { permanent_uuid: "7cc58d26c1154b37bd7f43bdc47d0449" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39233 } } peers { permanent_uuid: "a61795d2849947ef97be8e4b1709dddf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44263 } }
I20260619 16:50:55.787132 12133 raft_consensus.cc:374] T 6d956c8aaee84bfe98f7e86ba9e4cbac P a61795d2849947ef97be8e4b1709dddf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:55.787213 12133 raft_consensus.cc:729] T 6d956c8aaee84bfe98f7e86ba9e4cbac P a61795d2849947ef97be8e4b1709dddf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a61795d2849947ef97be8e4b1709dddf, State: Initialized, Role: FOLLOWER
I20260619 16:50:55.787460 12133 consensus_queue.cc:260] T 6d956c8aaee84bfe98f7e86ba9e4cbac P a61795d2849947ef97be8e4b1709dddf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e144352b42cf4e9aab14a5b74482a3c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46031 } } peers { permanent_uuid: "7cc58d26c1154b37bd7f43bdc47d0449" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39233 } } peers { permanent_uuid: "a61795d2849947ef97be8e4b1709dddf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44263 } }
I20260619 16:50:55.787927 12133 ts_tablet_manager.cc:1428] T 6d956c8aaee84bfe98f7e86ba9e4cbac P a61795d2849947ef97be8e4b1709dddf: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20260619 16:50:55.788163 12132 raft_consensus.cc:374] T 6d956c8aaee84bfe98f7e86ba9e4cbac P 7cc58d26c1154b37bd7f43bdc47d0449 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:50:55.788190 11842 heartbeater.cc:503] Master 127.0.98.62:40299 was elected leader, sending a full tablet report...
I20260619 16:50:55.788298 12132 raft_consensus.cc:729] T 6d956c8aaee84bfe98f7e86ba9e4cbac P 7cc58d26c1154b37bd7f43bdc47d0449 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cc58d26c1154b37bd7f43bdc47d0449, State: Initialized, Role: FOLLOWER
I20260619 16:50:55.788515 12132 consensus_queue.cc:260] T 6d956c8aaee84bfe98f7e86ba9e4cbac P 7cc58d26c1154b37bd7f43bdc47d0449 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e144352b42cf4e9aab14a5b74482a3c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46031 } } peers { permanent_uuid: "7cc58d26c1154b37bd7f43bdc47d0449" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39233 } } peers { permanent_uuid: "a61795d2849947ef97be8e4b1709dddf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44263 } }
I20260619 16:50:55.788892 12132 ts_tablet_manager.cc:1428] T 6d956c8aaee84bfe98f7e86ba9e4cbac P 7cc58d26c1154b37bd7f43bdc47d0449: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260619 16:50:55.789060 11980 heartbeater.cc:503] Master 127.0.98.62:40299 was elected leader, sending a full tablet report...
W20260619 16:50:55.797348 11843 tablet.cc:2369] T 6d956c8aaee84bfe98f7e86ba9e4cbac P a61795d2849947ef97be8e4b1709dddf: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 16:50:55 dist-test-slave-jsw9 krb5kdc[11153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260619 16:50:55.880749 11981 tablet.cc:2369] T 6d956c8aaee84bfe98f7e86ba9e4cbac P 7cc58d26c1154b37bd7f43bdc47d0449: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:50:55.936789 12115 tablet.cc:2369] T 6d956c8aaee84bfe98f7e86ba9e4cbac P e144352b42cf4e9aab14a5b74482a3c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:50:56.144577 12137 raft_consensus.cc:482] T 6d956c8aaee84bfe98f7e86ba9e4cbac P e144352b42cf4e9aab14a5b74482a3c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:50:56.144778 12137 raft_consensus.cc:504] T 6d956c8aaee84bfe98f7e86ba9e4cbac P e144352b42cf4e9aab14a5b74482a3c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e144352b42cf4e9aab14a5b74482a3c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46031 } } peers { permanent_uuid: "7cc58d26c1154b37bd7f43bdc47d0449" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39233 } } peers { permanent_uuid: "a61795d2849947ef97be8e4b1709dddf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44263 } }
I20260619 16:50:56.145304 12137 leader_election.cc:288] T 6d956c8aaee84bfe98f7e86ba9e4cbac P e144352b42cf4e9aab14a5b74482a3c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7cc58d26c1154b37bd7f43bdc47d0449 (127.0.98.2:39233), a61795d2849947ef97be8e4b1709dddf (127.0.98.1:44263)
I20260619 16:50:56.149959 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:56.145613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44263 (local address 127.0.98.3:42303)
0619 16:50:56.145736 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:56.145749 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:56.145804 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:56.146398 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:56.146400 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:56.146402 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:56.146492 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:56.146495 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:56.147612 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:56.147613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:56.148878 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:56.148882 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:56.149682 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:56.149683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:56.149766 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:56.149771 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:50:56.149812 (+ 41us) client_negotiation.cc:240] Negotiation successful
0619 16:50:56.149863 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260619 16:50:56.150558 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:56.145900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42303 (local address 127.0.98.1:44263)
0619 16:50:56.145949 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:56.145951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:56.145959 (+ 8us) server_negotiation.cc:407] Connection header received
0619 16:50:56.145991 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:56.145992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:56.146005 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:56.146059 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:56.146575 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:56.147348 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:56.148993 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:56.149572 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:56.149635 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:56.149650 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:56.150427 (+ 777us) server_negotiation.cc:299] Negotiation successful
0619 16:50:56.150485 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 16:50:56.150915 11796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d956c8aaee84bfe98f7e86ba9e4cbac" candidate_uuid: "e144352b42cf4e9aab14a5b74482a3c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a61795d2849947ef97be8e4b1709dddf" is_pre_election: true
I20260619 16:50:56.151158 11796 raft_consensus.cc:2454] T 6d956c8aaee84bfe98f7e86ba9e4cbac P a61795d2849947ef97be8e4b1709dddf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e144352b42cf4e9aab14a5b74482a3c5 in term 0.
I20260619 16:50:56.151456 12003 leader_election.cc:302] T 6d956c8aaee84bfe98f7e86ba9e4cbac P e144352b42cf4e9aab14a5b74482a3c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a61795d2849947ef97be8e4b1709dddf, e144352b42cf4e9aab14a5b74482a3c5; no voters:
I20260619 16:50:56.151625 12137 raft_consensus.cc:2781] T 6d956c8aaee84bfe98f7e86ba9e4cbac P e144352b42cf4e9aab14a5b74482a3c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:50:56.151684 12137 raft_consensus.cc:482] T 6d956c8aaee84bfe98f7e86ba9e4cbac P e144352b42cf4e9aab14a5b74482a3c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:50:56.151712 12137 raft_consensus.cc:3037] T 6d956c8aaee84bfe98f7e86ba9e4cbac P e144352b42cf4e9aab14a5b74482a3c5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:56.152863 12137 raft_consensus.cc:504] T 6d956c8aaee84bfe98f7e86ba9e4cbac P e144352b42cf4e9aab14a5b74482a3c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e144352b42cf4e9aab14a5b74482a3c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46031 } } peers { permanent_uuid: "7cc58d26c1154b37bd7f43bdc47d0449" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39233 } } peers { permanent_uuid: "a61795d2849947ef97be8e4b1709dddf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44263 } }
I20260619 16:50:56.153059 12137 leader_election.cc:288] T 6d956c8aaee84bfe98f7e86ba9e4cbac P e144352b42cf4e9aab14a5b74482a3c5 [CANDIDATE]: Term 1 election: Requested vote from peers 7cc58d26c1154b37bd7f43bdc47d0449 (127.0.98.2:39233), a61795d2849947ef97be8e4b1709dddf (127.0.98.1:44263)
I20260619 16:50:56.153288 11796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d956c8aaee84bfe98f7e86ba9e4cbac" candidate_uuid: "e144352b42cf4e9aab14a5b74482a3c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a61795d2849947ef97be8e4b1709dddf"
I20260619 16:50:56.153378 11796 raft_consensus.cc:3037] T 6d956c8aaee84bfe98f7e86ba9e4cbac P a61795d2849947ef97be8e4b1709dddf [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:56.153661 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:56.147476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39233 (local address 127.0.98.3:35305)
0619 16:50:56.150082 (+ 2606us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:56.150094 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:50:56.150137 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:50:56.150769 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:50:56.150771 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:50:56.150773 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:50:56.150809 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:56.150811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:56.151691 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:56.151692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:56.152703 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:50:56.152705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:50:56.153458 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:50:56.153459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:50:56.153512 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:56.153515 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:50:56.153555 (+ 40us) client_negotiation.cc:240] Negotiation successful
0619 16:50:56.153593 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2593,"thread_start_us":85,"threads_started":1}
I20260619 16:50:56.153853 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:56.146318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35305 (local address 127.0.98.2:39233)
0619 16:50:56.146362 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:56.146364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:56.150212 (+ 3848us) server_negotiation.cc:407] Connection header received
0619 16:50:56.150247 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:56.150248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:56.150264 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:56.150306 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:50:56.150889 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:56.151605 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:56.152796 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:56.153354 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:56.153416 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:56.153428 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:56.153734 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 16:50:56.153779 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 16:50:56.154371 11796 raft_consensus.cc:2454] T 6d956c8aaee84bfe98f7e86ba9e4cbac P a61795d2849947ef97be8e4b1709dddf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e144352b42cf4e9aab14a5b74482a3c5 in term 1.
I20260619 16:50:56.154620 12003 leader_election.cc:302] T 6d956c8aaee84bfe98f7e86ba9e4cbac P e144352b42cf4e9aab14a5b74482a3c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a61795d2849947ef97be8e4b1709dddf, e144352b42cf4e9aab14a5b74482a3c5; no voters:
I20260619 16:50:56.154757 12137 raft_consensus.cc:2781] T 6d956c8aaee84bfe98f7e86ba9e4cbac P e144352b42cf4e9aab14a5b74482a3c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:50:56.154794 11933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d956c8aaee84bfe98f7e86ba9e4cbac" candidate_uuid: "e144352b42cf4e9aab14a5b74482a3c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cc58d26c1154b37bd7f43bdc47d0449"
I20260619 16:50:56.154917 12137 raft_consensus.cc:686] T 6d956c8aaee84bfe98f7e86ba9e4cbac P e144352b42cf4e9aab14a5b74482a3c5 [term 1 LEADER]: Becoming Leader. State: Replica: e144352b42cf4e9aab14a5b74482a3c5, State: Running, Role: LEADER
I20260619 16:50:56.154930 11933 raft_consensus.cc:3037] T 6d956c8aaee84bfe98f7e86ba9e4cbac P 7cc58d26c1154b37bd7f43bdc47d0449 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:50:56.155025 12137 consensus_queue.cc:237] T 6d956c8aaee84bfe98f7e86ba9e4cbac P e144352b42cf4e9aab14a5b74482a3c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e144352b42cf4e9aab14a5b74482a3c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46031 } } peers { permanent_uuid: "7cc58d26c1154b37bd7f43bdc47d0449" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39233 } } peers { permanent_uuid: "a61795d2849947ef97be8e4b1709dddf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44263 } }
I20260619 16:50:56.155901 11933 raft_consensus.cc:2454] T 6d956c8aaee84bfe98f7e86ba9e4cbac P 7cc58d26c1154b37bd7f43bdc47d0449 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e144352b42cf4e9aab14a5b74482a3c5 in term 1.
I20260619 16:50:56.156111 11934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d956c8aaee84bfe98f7e86ba9e4cbac" candidate_uuid: "e144352b42cf4e9aab14a5b74482a3c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cc58d26c1154b37bd7f43bdc47d0449" is_pre_election: true
I20260619 16:50:56.156190 11934 raft_consensus.cc:2362] T 6d956c8aaee84bfe98f7e86ba9e4cbac P 7cc58d26c1154b37bd7f43bdc47d0449 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate e144352b42cf4e9aab14a5b74482a3c5 in term 1. Re-sending same reply.
I20260619 16:50:56.156502 11629 catalog_manager.cc:5495] T 6d956c8aaee84bfe98f7e86ba9e4cbac P e144352b42cf4e9aab14a5b74482a3c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to e144352b42cf4e9aab14a5b74482a3c5 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e144352b42cf4e9aab14a5b74482a3c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e144352b42cf4e9aab14a5b74482a3c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7cc58d26c1154b37bd7f43bdc47d0449" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a61795d2849947ef97be8e4b1709dddf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44263 } health_report { overall_health: UNKNOWN } } }
Jun 19 16:50:56 dist-test-slave-jsw9 krb5kdc[11153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:50:56.191951 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:56.187561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43312 (local address 127.0.98.3:46031)
0619 16:50:56.187757 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 16:50:56.187760 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:50:56.187771 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:50:56.187825 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:50:56.187826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:50:56.187855 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:50:56.187935 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:50:56.188138 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:56.188891 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:56.189922 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:50:56.190268 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:50:56.190329 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:50:56.191446 (+ 1117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:50:56.191630 (+ 184us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:50:56.191719 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 16:50:56.191796 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 16:50:56.191828 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260619 16:50:56.194567 11796 raft_consensus.cc:1261] T 6d956c8aaee84bfe98f7e86ba9e4cbac P a61795d2849947ef97be8e4b1709dddf [term 1 FOLLOWER]: Refusing update from remote peer e144352b42cf4e9aab14a5b74482a3c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:50:56.194568 11934 raft_consensus.cc:1261] T 6d956c8aaee84bfe98f7e86ba9e4cbac P 7cc58d26c1154b37bd7f43bdc47d0449 [term 1 FOLLOWER]: Refusing update from remote peer e144352b42cf4e9aab14a5b74482a3c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:50:56.194986 12137 consensus_queue.cc:1035] T 6d956c8aaee84bfe98f7e86ba9e4cbac P e144352b42cf4e9aab14a5b74482a3c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cc58d26c1154b37bd7f43bdc47d0449" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:50:56.195385 12142 consensus_queue.cc:1035] T 6d956c8aaee84bfe98f7e86ba9e4cbac P e144352b42cf4e9aab14a5b74482a3c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a61795d2849947ef97be8e4b1709dddf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:50:56.198719 12146 mvcc.cc:205] Tried to move back new op lower bound from 7298612658969513984 to 7298612658811924480. Current Snapshot: MvccSnapshot[applied={T|T < 7298612658969513984}]
I20260619 16:50:56.202057 12147 mvcc.cc:205] Tried to move back new op lower bound from 7298612658969513984 to 7298612658811924480. Current Snapshot: MvccSnapshot[applied={T|T < 7298612658969513984 or (T in {7298612658969513984})}]
I20260619 16:50:56.205070 11629 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:50:56.205227 11629 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:50:56.206568 11629 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65fadfeef38642428dc241e1852c7f24: Sending DeleteTablet for 3 replicas of tablet 6d956c8aaee84bfe98f7e86ba9e4cbac
I20260619 16:50:56.207194 11776 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d956c8aaee84bfe98f7e86ba9e4cbac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:50:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51402
I20260619 16:50:56.207525 11914 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d956c8aaee84bfe98f7e86ba9e4cbac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:50:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39816
I20260619 16:50:56.207970 12037 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d956c8aaee84bfe98f7e86ba9e4cbac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:50:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43304
I20260619 16:50:56.208920 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 11711
I20260619 16:50:56.214787 12148 mvcc.cc:205] Tried to move back new op lower bound from 7298612658969513984 to 7298612658811924480. Current Snapshot: MvccSnapshot[applied={T|T < 7298612658969513984}]
I20260619 16:50:56.227572 12156 tablet_replica.cc:330] T 6d956c8aaee84bfe98f7e86ba9e4cbac P e144352b42cf4e9aab14a5b74482a3c5: stopping tablet replica
I20260619 16:50:56.227799 12156 raft_consensus.cc:2229] T 6d956c8aaee84bfe98f7e86ba9e4cbac P e144352b42cf4e9aab14a5b74482a3c5 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:50:56.227960 12156 raft_consensus.cc:2258] T 6d956c8aaee84bfe98f7e86ba9e4cbac P e144352b42cf4e9aab14a5b74482a3c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:50:56.234359 12155 tablet_replica.cc:330] T 6d956c8aaee84bfe98f7e86ba9e4cbac P 7cc58d26c1154b37bd7f43bdc47d0449: stopping tablet replica
I20260619 16:50:56.234670 12155 raft_consensus.cc:2229] T 6d956c8aaee84bfe98f7e86ba9e4cbac P 7cc58d26c1154b37bd7f43bdc47d0449 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:50:56.235365 12156 ts_tablet_manager.cc:1905] T 6d956c8aaee84bfe98f7e86ba9e4cbac P e144352b42cf4e9aab14a5b74482a3c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:50:56.235673 12155 raft_consensus.cc:2258] T 6d956c8aaee84bfe98f7e86ba9e4cbac P 7cc58d26c1154b37bd7f43bdc47d0449 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:50:56.238659 12155 ts_tablet_manager.cc:1905] T 6d956c8aaee84bfe98f7e86ba9e4cbac P 7cc58d26c1154b37bd7f43bdc47d0449: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 16:50:56.241819 11616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44263 (error 108)
I20260619 16:50:56.243046 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:56.242893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44263 (local address 127.0.0.1:51410)
0619 16:50:56.242959 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:56.243000 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 16:50:56.243360 11616 catalog_manager.cc:4584] TS a61795d2849947ef97be8e4b1709dddf (127.0.98.1:44263): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6d956c8aaee84bfe98f7e86ba9e4cbac: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44263: connect: Connection refused (error 111)
I20260619 16:50:56.243669 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 11848
I20260619 16:50:56.245687 12156 ts_tablet_manager.cc:1918] T 6d956c8aaee84bfe98f7e86ba9e4cbac P e144352b42cf4e9aab14a5b74482a3c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:50:56.245810 12156 log.cc:1196] T 6d956c8aaee84bfe98f7e86ba9e4cbac P e144352b42cf4e9aab14a5b74482a3c5: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781887741903642-392-0/minicluster-data/ts-2/wal/wals/6d956c8aaee84bfe98f7e86ba9e4cbac
I20260619 16:50:56.246097 12156 ts_tablet_manager.cc:1939] T 6d956c8aaee84bfe98f7e86ba9e4cbac P e144352b42cf4e9aab14a5b74482a3c5: Deleting consensus metadata
I20260619 16:50:56.254823 11616 catalog_manager.cc:4841] TS e144352b42cf4e9aab14a5b74482a3c5 (127.0.98.3:46031): tablet 6d956c8aaee84bfe98f7e86ba9e4cbac (table test-table [id=6d14ba22849a4428b0ef26bbe3ac356f]) successfully deleted
I20260619 16:50:56.258993 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:50:56.258826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39233 (local address 127.0.0.1:39824)
0619 16:50:56.258893 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0619 16:50:56.258931 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 16:50:56.259994 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 11985
I20260619 16:50:56.266652 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 11598
I20260619 16:50:56.276978 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 16:50:56.627274 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 16:50:56.627417 392 mini_ranger.cc:66] Stopping Ranger...
I20260619 16:50:56.973709 392 mini_ranger.cc:68] Stopped Ranger
2026-06-19 16:50:56.974 UTC [11183] LOG: received smart shutdown request
2026-06-19 16:50:56.979 UTC [11183] LOG: background worker "logical replication launcher" (PID 11192) exited with exit code 1
2026-06-19 16:50:56.980 UTC [11187] LOG: shutting down
2026-06-19 16:50:57.167 UTC [11183] LOG: database system is shut down
2026-06-19T16:50:57Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (55393 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:50:57 dist-test-slave-jsw9 krb5kdc[12168](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:50:57 dist-test-slave-jsw9 krb5kdc[12168](info): set up 2 sockets
Jun 19 16:50:57 dist-test-slave-jsw9 krb5kdc[12168](info): commencing operation
krb5kdc: starting...
W20260619 16:50:58.258270 392 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:50:58 dist-test-slave-jsw9 krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:50:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:50:58Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260619 16:50:58.455088 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskjYBH_Y/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/postgres -l logfile start
2026-06-19 16:51:01.748 UTC [12198] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 16:51:01.748 UTC [12198] LOG: listening on IPv4 address "127.0.98.20", port 53871
2026-06-19 16:51:01.753 UTC [12198] LOG: listening on Unix socket "/tmp/.s.PGSQL.53871"
2026-06-19 16:51:01.782 UTC [12201] LOG: database system was shut down at 2026-06-19 16:50:59 UTC
2026-06-19 16:51:01.789 UTC [12198] LOG: database system is ready to accept connections
I20260619 16:51:02.682395 392 mini_postgres.cc:96] Postgres bound to 53871
2026-06-19 16:51:02.689 UTC [12209] FATAL: database "slave" does not exist
127.0.98.20:53871 - accepting connections
I20260619 16:51:02.690124 392 mini_ranger.cc:161] Starting Ranger...
I20260619 16:51:02.707831 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 16:51:03.345132 392 mini_ranger.cc:87] Created ranger Postgres database
I20260619 16:51:03.345214 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ranger-admin
2026-06-19 16:51:03,637 [I] DB FLAVOR :POSTGRES
2026-06-19 16:51:03,638 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 16:51:03,638 [I] Checking connection..
2026-06-19 16:51:03,638 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 16:51:03,944 [I] Checking connection passed.
2026-06-19 16:51:03,945 [I] --------- Verifying version history table ---------
2026-06-19 16:51:03,945 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:51:04,263 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 16:51:04,264 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:51:04,578 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 16:51:04,579 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 16:51:04,579 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53871/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 16:51:04.879 UTC [12273] WARNING: there is no transaction in progress
2026-06-19 16:51:04,886 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 16:51:04,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:51:05,188 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 16:51:05,188 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 16:51:05,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 16:51:05,462 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 16:51:05,769 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-jsw9.c.gcp-upstream.internal','N') ;"
2026-06-19 16:51:06,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 16:51:06,349 [I] Table x_portal_user does not exist in database ranger
2026-06-19 16:51:06,349 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 16:51:06,664 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 16:51:06,664 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 16:51:06,972 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 16:51:07,284 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 16:51:07,285 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53871/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 16:51:08.526 UTC [12381] WARNING: there is no transaction in progress
2026-06-19 16:51:08.547 UTC [12381] WARNING: there is no transaction in progress
2026-06-19 16:51:08.568 UTC [12381] WARNING: there is no transaction in progress
2026-06-19 16:51:08.590 UTC [12381] WARNING: there is no transaction in progress
2026-06-19 16:51:08.616 UTC [12381] WARNING: there is no transaction in progress
2026-06-19 16:51:08.639 UTC [12381] WARNING: there is no transaction in progress
2026-06-19 16:51:08.719 UTC [12381] WARNING: there is no transaction in progress
2026-06-19 16:51:08.730 UTC [12381] WARNING: there is no transaction in progress
2026-06-19 16:51:08.754 UTC [12381] WARNING: there is no transaction in progress
2026-06-19 16:51:08.767 UTC [12381] WARNING: there is no transaction in progress
2026-06-19 16:51:08.781 UTC [12381] WARNING: there is no transaction in progress
2026-06-19 16:51:08.798 UTC [12381] WARNING: there is no transaction in progress
2026-06-19 16:51:08.810 UTC [12381] WARNING: there is no transaction in progress
2026-06-19 16:51:08.819 UTC [12381] WARNING: there is no transaction in progress
2026-06-19 16:51:08.827 UTC [12381] WARNING: there is no transaction in progress
2026-06-19 16:51:08.835 UTC [12381] WARNING: there is no transaction in progress
2026-06-19 16:51:09,767 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 16:51:09,767 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 16:51:10,033 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 16:51:10,034 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 16:51:10,336 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 16:51:10,336 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 16:51:10,664 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 16:51:10,664 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 16:51:10,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 16:51:11,233 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-jsw9.c.gcp-upstream.internal';"
2026-06-19 16:51:11,497 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 16:51:11,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 16:51:11,783 [I] DB_PATCHES have already been applied
I20260619 16:51:11.790086 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskjYBH_Y/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 16:51:11.790180 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260619 16:51:11.794586 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:60105
Jun 19, 2026 4:51:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 4:51:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 4:51:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 4:51:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 4:51:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 4:51:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 4:51:12 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 16:51:13.884467 392 mini_ranger.cc:160] Time spent starting Ranger: real 11.194s user 0.002s sys 0.011s
Jun 19, 2026 4:51:36 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 4:51:36 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 4:51:36 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 4:51:36 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 4:51:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 4:51:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 4:51:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 4:51:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 4:51:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 4:51:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 4:51:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 4:51:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 4:51:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 4:51:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 4:51:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 4:51:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 4:51:36 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 4:51:37 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 16:51:38.390697 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 16:51:38 dist-test-slave-jsw9 krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887898, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 16:51:38.487484 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 16:51:38.500504 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 16:51:39.592958 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 16:51:39.593052 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ranger-kms
I20260619 16:51:39.593070 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:53871
2026-06-19 16:51:39,799 [I] DB FLAVOR :POSTGRES
2026-06-19 16:51:39,800 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 16:51:39,800 [I] Checking connection
2026-06-19 16:51:39,800 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53871/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 16:51:40,128 [I] connection success
2026-06-19 16:51:40,129 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 16:51:40,129 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 16:51:40,129 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53871/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 16:51:40,449 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 16:51:40,450 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 16:51:40,450 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 16:51:40,450 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53871/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 16:51:40,800 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 16:51:41.405936 392 mini_ranger_kms.cc:310] Created kms service
I20260619 16:51:41.652158 392 mini_ranger_kms.cc:326] Created kudu user
I20260619 16:51:41.710441 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 16:51:41 dist-test-slave-jsw9 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887898, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:51:42.175243 392 mini_ranger_kms.cc:384] Added ranger policy
I20260619 16:51:42.175411 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskjYBH_Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskjYBH_Y/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 16:51:42.175433 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260619 16:51:42.179840 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12563
I20260619 16:51:42.179939 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:36039
Jun 19, 2026 4:51:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 4:51:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 4:51:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 4:51:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 4:51:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 4:51:43 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 16:51:44.253974 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.766s user 0.012s sys 0.174s
I20260619 16:51:44.254071 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 16:51:44.254112 392 mini_ranger_kms.cc:401] 127.0.98.20:36039/kms/v1/keys
Jun 19 16:51:44 dist-test-slave-jsw9 krb5kdc[12168](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781887904, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 16:51:45 dist-test-slave-jsw9 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887904, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 16:51:45 dist-test-slave-jsw9 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887904, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 16:51:48 dist-test-slave-jsw9 krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:51:48.059839 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41653
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41653
--ranger_config_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36039/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:51:48.151001 12613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:51:48.151247 12613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:51:48.151301 12613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:51:48.152531 12613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:51:48.152581 12613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:51:48.152599 12613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:51:48.152616 12613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:51:48.152630 12613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:51:48.154390 12613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:36039/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41653
--ranger_config_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:51:48.154668 12613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:51:48.154904 12613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:51:48.160548 12618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:51:48.160624 12613 server_base.cc:1034] running on GCE node
W20260619 16:51:48.160557 12619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:51:48.160578 12621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:51:48.161011 12613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:51:48.161339 12613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:51:48.162480 12613 hybrid_clock.cc:648] HybridClock initialized: now 1781887908162465 us; error 33 us; skew 500 ppm
Jun 19 16:51:48 dist-test-slave-jsw9 krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:51:48.165230 12613 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:51:48.165736 12613 webserver.cc:460] Webserver started at http://127.0.98.62:37379/ using document root <none> and password file <none>
I20260619 16:51:48.165951 12613 fs_manager.cc:359] Metadata directory not provided
I20260619 16:51:48.165987 12613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:51:48.166098 12613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:51:48 dist-test-slave-jsw9 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:51:48.219725 12613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "7b09c65a3cda4ed380f2ba7e5dd6e2be"
format_stamp: "Formatted at 2026-06-19 16:51:48 on dist-test-slave-jsw9"
server_key: "78761c3d871f1c7d40a65cb511ced3d8"
server_key_iv: "cd4e88cced92afcc7bad87a004c1f5e9"
server_key_version: "kuduclusterkey@0"
I20260619 16:51:48.220162 12613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "7b09c65a3cda4ed380f2ba7e5dd6e2be"
format_stamp: "Formatted at 2026-06-19 16:51:48 on dist-test-slave-jsw9"
server_key: "78761c3d871f1c7d40a65cb511ced3d8"
server_key_iv: "cd4e88cced92afcc7bad87a004c1f5e9"
server_key_version: "kuduclusterkey@0"
I20260619 16:51:48.221645 12613 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 16:51:48.235697 12628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:48.235934 12613 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:51:48.236064 12613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "7b09c65a3cda4ed380f2ba7e5dd6e2be"
format_stamp: "Formatted at 2026-06-19 16:51:48 on dist-test-slave-jsw9"
server_key: "78761c3d871f1c7d40a65cb511ced3d8"
server_key_iv: "cd4e88cced92afcc7bad87a004c1f5e9"
server_key_version: "kuduclusterkey@0"
I20260619 16:51:48.236171 12613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:51:48.249336 12613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:51:48.251477 12613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:51:48.251638 12613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:51:48.256904 12613 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41653
I20260619 16:51:48.257426 12613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:51:48.257826 12680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41653 every 8 connection(s)
I20260619 16:51:48.258273 12681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:51:48.260915 12681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be: Bootstrap starting.
I20260619 16:51:48.261600 12681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be: Neither blocks nor log segments found. Creating new log.
I20260619 16:51:48.261914 12681 log.cc:824] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be: Log is configured to *not* fsync() on all Append() calls
I20260619 16:51:48.262763 12681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be: No bootstrap required, opened a new log
I20260619 16:51:48.265012 12681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b09c65a3cda4ed380f2ba7e5dd6e2be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41653 } }
I20260619 16:51:48.265183 12681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:51:48.265206 12681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b09c65a3cda4ed380f2ba7e5dd6e2be, State: Initialized, Role: FOLLOWER
I20260619 16:51:48.265354 12681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b09c65a3cda4ed380f2ba7e5dd6e2be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41653 } }
I20260619 16:51:48.265431 12681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:51:48.265462 12681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:51:48.265511 12681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:51:48.266144 12681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b09c65a3cda4ed380f2ba7e5dd6e2be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41653 } }
I20260619 16:51:48.266274 12681 leader_election.cc:302] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b09c65a3cda4ed380f2ba7e5dd6e2be; no voters:
I20260619 16:51:48.266443 12681 leader_election.cc:288] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:51:48.266470 12684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:51:48.266659 12684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be [term 1 LEADER]: Becoming Leader. State: Replica: 7b09c65a3cda4ed380f2ba7e5dd6e2be, State: Running, Role: LEADER
I20260619 16:51:48.266755 12681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:51:48.266815 12684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b09c65a3cda4ed380f2ba7e5dd6e2be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41653 } }
I20260619 16:51:48.267177 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 12613
I20260619 16:51:48.267277 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:51:48.267233 12684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b09c65a3cda4ed380f2ba7e5dd6e2be. Latest consensus state: current_term: 1 leader_uuid: "7b09c65a3cda4ed380f2ba7e5dd6e2be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b09c65a3cda4ed380f2ba7e5dd6e2be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41653 } } }
I20260619 16:51:48.267305 12684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be [sys.catalog]: This master's current role is: LEADER
I20260619 16:51:48.267282 12686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b09c65a3cda4ed380f2ba7e5dd6e2be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b09c65a3cda4ed380f2ba7e5dd6e2be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41653 } } }
I20260619 16:51:48.267381 12686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be [sys.catalog]: This master's current role is: LEADER
I20260619 16:51:48.267505 392 external_mini_cluster.cc:1428] Setting key 525c3617ad3536576a8c769f3be4f9f2
I20260619 16:51:48.267540 12688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:51:48.268321 12688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:51:48.269752 12681 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 16:51:48.270151 12688 catalog_manager.cc:1347] Generated new cluster ID: 86ad6ae83a0f428b9dd7fad1f112181b
I20260619 16:51:48.270278 12688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:51:48.281487 12688 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:51:48.282407 12688 catalog_manager.cc:1504] Loading token signing keys...
Jun 19 16:51:48 dist-test-slave-jsw9 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:51:48.289525 12688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be: Generated new TSK 0
I20260619 16:51:48.289834 12688 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:51:48.292312 12692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:48.268027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33226 (local address 127.0.98.62:41653)
0619 16:51:48.278559 (+ 10532us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:48.278587 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:48.278669 (+ 82us) server_negotiation.cc:407] Connection header received
0619 16:51:48.279049 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:48.279060 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:48.279150 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:48.279375 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:48.280301 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:48.281952 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:48.283050 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:48.283760 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:48.283893 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:48.287572 (+ 3679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:48.287579 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:48.287580 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:48.287596 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:48.290072 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:48.290554 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:48.290558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:48.290558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:48.290674 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:48.290867 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:48.290868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:48.290869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:48.291198 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:48.291374 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:48.291847 (+ 473us) server_negotiation.cc:299] Negotiation successful
0619 16:51:48.292025 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10373,"thread_start_us":96,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:51:49.139420 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36039/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41653
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:51:49.262424 12726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:51:49.262670 12726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:51:49.262706 12726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:51:49.264015 12726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:51:49.264060 12726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:51:49.264098 12726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:51:49.265655 12726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:36039/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:51:49.265923 12726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:51:49.266175 12726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:51:49.272655 12734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:51:49.272727 12731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:51:49.272938 12732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:51:49.273041 12726 server_base.cc:1034] running on GCE node
I20260619 16:51:49.273254 12726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:51:49.273499 12726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:51:49.274665 12726 hybrid_clock.cc:648] HybridClock initialized: now 1781887909274645 us; error 35 us; skew 500 ppm
Jun 19 16:51:49 dist-test-slave-jsw9 krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:51:49.277707 12726 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:51:49.278364 12726 webserver.cc:460] Webserver started at http://127.0.98.1:41691/ using document root <none> and password file <none>
I20260619 16:51:49.278604 12726 fs_manager.cc:359] Metadata directory not provided
I20260619 16:51:49.278653 12726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:51:49.278810 12726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:51:49 dist-test-slave-jsw9 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:51:49.298173 12726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-0/data/instance:
uuid: "51c7f6f7290143f1939e915dfdd50453"
format_stamp: "Formatted at 2026-06-19 16:51:49 on dist-test-slave-jsw9"
server_key: "40410573536570412e42f7ac0d67ea17"
server_key_iv: "535ab96971f316261e09d9877a1c5174"
server_key_version: "kuduclusterkey@0"
I20260619 16:51:49.299530 12726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-0/wal/instance:
uuid: "51c7f6f7290143f1939e915dfdd50453"
format_stamp: "Formatted at 2026-06-19 16:51:49 on dist-test-slave-jsw9"
server_key: "40410573536570412e42f7ac0d67ea17"
server_key_iv: "535ab96971f316261e09d9877a1c5174"
server_key_version: "kuduclusterkey@0"
I20260619 16:51:49.301717 12726 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:51:49.325635 12743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:49.325901 12726 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.001s
I20260619 16:51:49.326020 12726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-0/wal
uuid: "51c7f6f7290143f1939e915dfdd50453"
format_stamp: "Formatted at 2026-06-19 16:51:49 on dist-test-slave-jsw9"
server_key: "40410573536570412e42f7ac0d67ea17"
server_key_iv: "535ab96971f316261e09d9877a1c5174"
server_key_version: "kuduclusterkey@0"
I20260619 16:51:49.326159 12726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:51:49.340019 12726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:51:49.342096 12726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:51:49.342276 12726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:51:49.342550 12726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:51:49.342952 12726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:51:49.342989 12726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:49.343024 12726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:51:49.343042 12726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:49.351204 12726 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34241
I20260619 16:51:49.351657 12726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:51:49.352286 12856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34241 every 8 connection(s)
Jun 19 16:51:49 dist-test-slave-jsw9 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:51:49.358433 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 12726
I20260619 16:51:49.358523 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:51:49.358762 392 external_mini_cluster.cc:1428] Setting key 6a6b2f59794f5a6b0468dd86274dc03d
I20260619 16:51:49.363015 12860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:49.352530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41653 (local address 127.0.98.1:58675)
0619 16:51:49.352691 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:49.352740 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:49.353437 (+ 697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:49.353883 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:49.353888 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:49.354122 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:51:49.354431 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:49.354438 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:49.355230 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:49.355231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:49.356112 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:49.356114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:49.356527 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:49.356529 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:49.356657 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:49.357232 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:51:49.357243 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:51:49.358960 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:51:49.361046 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:49.361059 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:49.361062 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:49.361419 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:49.361672 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:49.361673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:49.361673 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:49.361855 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:49.362162 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:51:49.362170 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:51:49.362428 (+ 258us) client_negotiation.cc:769] Sending connection context
0619 16:51:49.362550 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 16:51:49.362760 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260619 16:51:49.363564 12859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:49.352537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58675 (local address 127.0.98.62:41653)
0619 16:51:49.353631 (+ 1094us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:49.353633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:49.353641 (+ 8us) server_negotiation.cc:407] Connection header received
0619 16:51:49.353669 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:49.353670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:49.353683 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:49.353755 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:49.354538 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:49.355127 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:49.356206 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:49.356434 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:49.356494 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:49.359084 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:49.359086 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:49.359086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:49.359095 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:49.360941 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:49.361516 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:49.361518 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:49.361519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:49.361602 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:49.361921 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:49.361922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:49.361923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:49.362080 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:49.362138 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:49.363373 (+ 1235us) server_negotiation.cc:299] Negotiation successful
0619 16:51:49.363481 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1066,"thread_start_us":70,"threads_started":1}
I20260619 16:51:49.363965 12857 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41653
I20260619 16:51:49.364090 12857 heartbeater.cc:464] Registering TS with master...
I20260619 16:51:49.364313 12857 heartbeater.cc:511] Master 127.0.98.62:41653 requested a full tablet report, sending...
I20260619 16:51:49.364923 12645 ts_manager.cc:194] Registered new tserver with Master: 51c7f6f7290143f1939e915dfdd50453 (127.0.98.1:34241)
I20260619 16:51:49.365602 12645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58675
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:51:49.393362 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36039/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41653
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:51:49.502682 12863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:51:49.502928 12863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:51:49.502966 12863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:51:49.504426 12863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:51:49.504482 12863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:51:49.504522 12863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:51:49.506525 12863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:36039/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:51:49.506816 12863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:51:49.507120 12863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:51:49.512988 12873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:51:49.513010 12871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:51:49.513103 12870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:51:49.513283 12863 server_base.cc:1034] running on GCE node
I20260619 16:51:49.513491 12863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:51:49.513782 12863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:51:49.514966 12863 hybrid_clock.cc:648] HybridClock initialized: now 1781887909514942 us; error 35 us; skew 500 ppm
Jun 19 16:51:49 dist-test-slave-jsw9 krb5kdc[12168](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781887909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 16:51:49 dist-test-slave-jsw9 krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:51:49.518396 12863 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:51:49.519088 12863 webserver.cc:460] Webserver started at http://127.0.98.2:38507/ using document root <none> and password file <none>
I20260619 16:51:49.519429 12863 fs_manager.cc:359] Metadata directory not provided
I20260619 16:51:49.519508 12863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:51:49.519675 12863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:51:49 dist-test-slave-jsw9 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:51:49.537192 12863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-1/data/instance:
uuid: "49734114abe9465b8382bf37e50a58c3"
format_stamp: "Formatted at 2026-06-19 16:51:49 on dist-test-slave-jsw9"
server_key: "9285e511203d04a096aac41f8dba4562"
server_key_iv: "2527e40486c676c2a693a2ba97e34eaa"
server_key_version: "kuduclusterkey@0"
I20260619 16:51:49.537889 12863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-1/wal/instance:
uuid: "49734114abe9465b8382bf37e50a58c3"
format_stamp: "Formatted at 2026-06-19 16:51:49 on dist-test-slave-jsw9"
server_key: "9285e511203d04a096aac41f8dba4562"
server_key_iv: "2527e40486c676c2a693a2ba97e34eaa"
server_key_version: "kuduclusterkey@0"
I20260619 16:51:49.542676 12863 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.000s
I20260619 16:51:49.562995 12880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:49.563251 12863 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260619 16:51:49.563390 12863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-1/wal
uuid: "49734114abe9465b8382bf37e50a58c3"
format_stamp: "Formatted at 2026-06-19 16:51:49 on dist-test-slave-jsw9"
server_key: "9285e511203d04a096aac41f8dba4562"
server_key_iv: "2527e40486c676c2a693a2ba97e34eaa"
server_key_version: "kuduclusterkey@0"
I20260619 16:51:49.563547 12863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:51:49.588485 12863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:51:49.591225 12863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:51:49.591423 12863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:51:49.591718 12863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:51:49.592149 12863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:51:49.592200 12863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:49.592238 12863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:51:49.592257 12863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:49.600544 12863 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44409
I20260619 16:51:49.601096 12863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:51:49.601742 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 12863
I20260619 16:51:49.601845 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:51:49.602056 392 external_mini_cluster.cc:1428] Setting key b8afcf3b0a172e8abc80ee35a7906f48
I20260619 16:51:49.603348 12994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44409 every 8 connection(s)
Jun 19 16:51:49 dist-test-slave-jsw9 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 16:51:49.615365 12997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:49.602931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41653 (local address 127.0.98.2:40047)
0619 16:51:49.603516 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:49.603567 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:49.604360 (+ 793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:49.604772 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:49.604778 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:49.605060 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:51:49.605405 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:49.605412 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:49.606240 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:49.606241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:49.607779 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:49.607782 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:49.608412 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:49.608413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:49.608573 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:49.609234 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:51:49.609247 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:51:49.611199 (+ 1952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:51:49.613150 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:49.613161 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:49.613165 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:49.613540 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:49.613789 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:49.613790 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:49.613790 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:49.613949 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:49.614459 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:51:49.614467 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:51:49.614703 (+ 236us) client_negotiation.cc:769] Sending connection context
0619 16:51:49.614832 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 16:51:49.615040 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":104,"threads_started":1}
I20260619 16:51:49.615891 12859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:49.602477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40047 (local address 127.0.98.62:41653)
0619 16:51:49.602540 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:49.602542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:49.603667 (+ 1125us) server_negotiation.cc:407] Connection header received
0619 16:51:49.604499 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:49.604500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:49.604522 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:49.604553 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:49.605517 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:49.606084 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:49.607904 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:49.608235 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:49.608315 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:49.611342 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:49.611344 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:49.611344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:49.611355 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:49.613040 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:49.613634 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:49.613635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:49.613636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:49.613718 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:49.614013 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:49.614014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:49.614015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:49.614345 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:49.614428 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:49.615672 (+ 1244us) server_negotiation.cc:299] Negotiation successful
0619 16:51:49.615789 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 16:51:49.616302 12995 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41653
I20260619 16:51:49.616425 12995 heartbeater.cc:464] Registering TS with master...
I20260619 16:51:49.616758 12995 heartbeater.cc:511] Master 127.0.98.62:41653 requested a full tablet report, sending...
I20260619 16:51:49.617245 12645 ts_manager.cc:194] Registered new tserver with Master: 49734114abe9465b8382bf37e50a58c3 (127.0.98.2:44409)
I20260619 16:51:49.617811 12645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40047
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:51:49.641187 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36039/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41653
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:51:49.778077 13000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:51:49.778656 13000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:51:49.778769 13000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:51:49.781145 13000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:51:49.781279 13000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:51:49.781368 13000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:51:49.784099 13000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:36039/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:51:49.784598 13000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:51:49.785334 13000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:51:49.793576 13006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:51:49.793701 13005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:51:49.793943 13000 server_base.cc:1034] running on GCE node
W20260619 16:51:49.793578 13008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:51:49.794330 13000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:51:49.794704 13000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:51:49.795871 13000 hybrid_clock.cc:648] HybridClock initialized: now 1781887909795845 us; error 47 us; skew 500 ppm
Jun 19 16:51:49 dist-test-slave-jsw9 krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:51:49.799088 13000 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:51:49.799765 13000 webserver.cc:460] Webserver started at http://127.0.98.3:43915/ using document root <none> and password file <none>
I20260619 16:51:49.800083 13000 fs_manager.cc:359] Metadata directory not provided
I20260619 16:51:49.800158 13000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:51:49.800318 13000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:51:49 dist-test-slave-jsw9 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:51:49.826074 13000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-2/data/instance:
uuid: "bf3f1e0cc2184097a0b0c8db60d890fc"
format_stamp: "Formatted at 2026-06-19 16:51:49 on dist-test-slave-jsw9"
server_key: "a98f3624bcaf0d3bbf12f5eca41011be"
server_key_iv: "0eebf98ff73595056b2ef91647af17e6"
server_key_version: "kuduclusterkey@0"
I20260619 16:51:49.826627 13000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bf3f1e0cc2184097a0b0c8db60d890fc"
format_stamp: "Formatted at 2026-06-19 16:51:49 on dist-test-slave-jsw9"
server_key: "a98f3624bcaf0d3bbf12f5eca41011be"
server_key_iv: "0eebf98ff73595056b2ef91647af17e6"
server_key_version: "kuduclusterkey@0"
I20260619 16:51:49.828604 13000 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:51:49.846338 13015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:49.846552 13000 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:51:49.846650 13000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-2/wal
uuid: "bf3f1e0cc2184097a0b0c8db60d890fc"
format_stamp: "Formatted at 2026-06-19 16:51:49 on dist-test-slave-jsw9"
server_key: "a98f3624bcaf0d3bbf12f5eca41011be"
server_key_iv: "0eebf98ff73595056b2ef91647af17e6"
server_key_version: "kuduclusterkey@0"
I20260619 16:51:49.846766 13000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:51:49.866166 13000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:51:49.877616 13000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:51:49.877851 13000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:51:49.878227 13000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:51:49.878718 13000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:51:49.878770 13000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:49.878810 13000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:51:49.878825 13000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:49.887333 13000 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38927
I20260619 16:51:49.887586 13128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38927 every 8 connection(s)
I20260619 16:51:49.887887 13000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:51:49.889528 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 13000
I20260619 16:51:49.889643 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:51:49.889873 392 external_mini_cluster.cc:1428] Setting key 83a51c0e968527119538dfc68e3a3b94
Jun 19 16:51:49 dist-test-slave-jsw9 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:51:49.902453 12859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:49.888772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56411 (local address 127.0.98.62:41653)
0619 16:51:49.888829 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:49.888832 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:49.889134 (+ 302us) server_negotiation.cc:407] Connection header received
0619 16:51:49.890984 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:49.890985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:49.890999 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:49.891029 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:49.892158 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:49.893005 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:49.894634 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:49.894977 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:49.895021 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:49.898129 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:49.898131 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:49.898131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:49.898142 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:49.900006 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:49.900591 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:49.900593 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:49.900594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:49.900669 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:49.901460 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:49.901461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:49.901462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:49.901635 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:49.901686 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:49.902219 (+ 533us) server_negotiation.cc:299] Negotiation successful
0619 16:51:49.902336 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 16:51:49.903003 13131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:49.888787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41653 (local address 127.0.98.3:56411)
0619 16:51:49.888981 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:49.889029 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:49.890864 (+ 1835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:49.891259 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:49.891265 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:49.891661 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:51:49.892030 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:49.892037 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:49.893130 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:49.893131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:49.894527 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:49.894531 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:49.895137 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:49.895138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:49.895294 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:49.896122 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:51:49.896134 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:51:49.897985 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:51:49.900115 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:49.900126 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:49.900129 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:49.900498 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:49.900742 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:49.900742 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:49.900743 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:49.901246 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:49.901772 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:51:49.901779 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:51:49.902019 (+ 240us) client_negotiation.cc:769] Sending connection context
0619 16:51:49.902147 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 16:51:49.902385 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":99,"threads_started":1}
I20260619 16:51:49.903501 13129 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41653
I20260619 16:51:49.903622 13129 heartbeater.cc:464] Registering TS with master...
I20260619 16:51:49.903868 13129 heartbeater.cc:511] Master 127.0.98.62:41653 requested a full tablet report, sending...
I20260619 16:51:49.904315 12645 ts_manager.cc:194] Registered new tserver with Master: bf3f1e0cc2184097a0b0c8db60d890fc (127.0.98.3:38927)
I20260619 16:51:49.904858 12645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56411
I20260619 16:51:49.912755 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:51:49.913676 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 12726
I20260619 16:51:49.920773 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 12863
I20260619 16:51:49.927767 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 13000
I20260619 16:51:49.934993 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 12613
I20260619 16:51:49.940897 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41653
--webserver_interface=127.0.98.62
--webserver_port=37379
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41653
--ranger_config_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36039/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36039/kms with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:51:50.039640 13132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:51:50.039902 13132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:51:50.039929 13132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:51:50.041193 13132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:51:50.041234 13132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:51:50.041247 13132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:51:50.041258 13132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:51:50.041270 13132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:51:50.043036 13132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36039/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41653
--ranger_config_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:51:50.043293 13132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:51:50.043510 13132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:51:50.049855 13138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:51:50.049835 13140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:51:50.049835 13137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:51:50.050403 13132 server_base.cc:1034] running on GCE node
I20260619 16:51:50.050683 13132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:51:50.051087 13132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:51:50.052279 13132 hybrid_clock.cc:648] HybridClock initialized: now 1781887910052251 us; error 50 us; skew 500 ppm
Jun 19 16:51:50 dist-test-slave-jsw9 krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:51:50.055684 13132 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:51:50.056232 13132 webserver.cc:460] Webserver started at http://127.0.98.62:37379/ using document root <none> and password file <none>
I20260619 16:51:50.056476 13132 fs_manager.cc:359] Metadata directory not provided
I20260619 16:51:50.056535 13132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 16:51:50 dist-test-slave-jsw9 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:51:50.135623 13132 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:51:50.136405 13148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:50.136584 13132 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:51:50.136679 13132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "7b09c65a3cda4ed380f2ba7e5dd6e2be"
format_stamp: "Formatted at 2026-06-19 16:51:48 on dist-test-slave-jsw9"
server_key: "78761c3d871f1c7d40a65cb511ced3d8"
server_key_iv: "cd4e88cced92afcc7bad87a004c1f5e9"
server_key_version: "kuduclusterkey@0"
I20260619 16:51:50.137151 13132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:51:50.149717 13132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:51:50.151728 13132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:51:50.151892 13132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:51:50.156646 13132 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41653
I20260619 16:51:50.156703 13200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41653 every 8 connection(s)
I20260619 16:51:50.157111 13132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:51:50.158645 13201 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:51:50.158771 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 13132
I20260619 16:51:50.159045 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36039/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34241
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41653
--webserver_port=41691
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36039/kms with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:51:50.159492 13201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be: Bootstrap starting.
I20260619 16:51:50.169157 13201 log.cc:824] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be: Log is configured to *not* fsync() on all Append() calls
I20260619 16:51:50.171200 13201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:51:50.171468 13201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be: Bootstrap complete.
I20260619 16:51:50.174005 13201 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b09c65a3cda4ed380f2ba7e5dd6e2be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41653 } }
I20260619 16:51:50.174317 13201 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b09c65a3cda4ed380f2ba7e5dd6e2be, State: Initialized, Role: FOLLOWER
I20260619 16:51:50.174508 13201 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b09c65a3cda4ed380f2ba7e5dd6e2be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41653 } }
I20260619 16:51:50.174589 13201 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:51:50.174640 13201 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:51:50.174696 13201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:51:50.175891 13201 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b09c65a3cda4ed380f2ba7e5dd6e2be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41653 } }
I20260619 16:51:50.175999 13201 leader_election.cc:302] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b09c65a3cda4ed380f2ba7e5dd6e2be; no voters:
I20260619 16:51:50.176182 13201 leader_election.cc:288] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 16:51:50.176246 13205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be [term 2 FOLLOWER]: Leader election won for term 2
I20260619 16:51:50.176362 13205 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be [term 2 LEADER]: Becoming Leader. State: Replica: 7b09c65a3cda4ed380f2ba7e5dd6e2be, State: Running, Role: LEADER
I20260619 16:51:50.176452 13205 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b09c65a3cda4ed380f2ba7e5dd6e2be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41653 } }
I20260619 16:51:50.176484 13201 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:51:50.176781 13206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7b09c65a3cda4ed380f2ba7e5dd6e2be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b09c65a3cda4ed380f2ba7e5dd6e2be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41653 } } }
I20260619 16:51:50.176793 13207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b09c65a3cda4ed380f2ba7e5dd6e2be. Latest consensus state: current_term: 2 leader_uuid: "7b09c65a3cda4ed380f2ba7e5dd6e2be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b09c65a3cda4ed380f2ba7e5dd6e2be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41653 } } }
I20260619 16:51:50.176898 13206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be [sys.catalog]: This master's current role is: LEADER
I20260619 16:51:50.176904 13207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be [sys.catalog]: This master's current role is: LEADER
I20260619 16:51:50.177104 13209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:51:50.177682 13209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:51:50.177860 13209 catalog_manager.cc:1259] Loaded cluster ID: 86ad6ae83a0f428b9dd7fad1f112181b
I20260619 16:51:50.177899 13209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:51:50.178256 13201 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 16:51:50.182072 13209 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:51:50.182723 13209 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be: Loaded TSK: 0
I20260619 16:51:50.183022 13209 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 16:51:50.281744 13203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:51:50.282088 13203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:51:50.282131 13203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:51:50.284253 13203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:51:50.284349 13203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:51:50.284401 13203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:51:50.287076 13203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36039/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41691
--tserver_master_addrs=127.0.98.62:41653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:51:50.287417 13203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:51:50.287742 13203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:51:50.297472 13241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:51:50.297863 13239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:51:50.298645 13238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:51:50.298885 13203 server_base.cc:1034] running on GCE node
I20260619 16:51:50.299149 13203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:51:50.299528 13203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:51:50.300727 13203 hybrid_clock.cc:648] HybridClock initialized: now 1781887910300699 us; error 41 us; skew 500 ppm
Jun 19 16:51:50 dist-test-slave-jsw9 krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:51:50.304410 13203 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:51:50.305089 13203 webserver.cc:460] Webserver started at http://127.0.98.1:41691/ using document root <none> and password file <none>
I20260619 16:51:50.305426 13203 fs_manager.cc:359] Metadata directory not provided
I20260619 16:51:50.305521 13203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 16:51:50 dist-test-slave-jsw9 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:51:50.335732 13203 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.001s sys 0.000s
I20260619 16:51:50.336851 13249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:50.337018 13203 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:51:50.337108 13203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-0/wal
uuid: "51c7f6f7290143f1939e915dfdd50453"
format_stamp: "Formatted at 2026-06-19 16:51:49 on dist-test-slave-jsw9"
server_key: "40410573536570412e42f7ac0d67ea17"
server_key_iv: "535ab96971f316261e09d9877a1c5174"
server_key_version: "kuduclusterkey@0"
I20260619 16:51:50.337685 13203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:51:50.359477 13203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:51:50.361958 13203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:51:50.362133 13203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:51:50.362476 13203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:51:50.362905 13203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:51:50.362952 13203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:50.362988 13203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:51:50.363004 13203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:50.373034 13203 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34241
I20260619 16:51:50.373594 13203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:51:50.375005 13362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34241 every 8 connection(s)
I20260619 16:51:50.378701 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 13203
I20260619 16:51:50.379055 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36039/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44409
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41653
--webserver_port=38507
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36039/kms with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:51:50 dist-test-slave-jsw9 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:51:50.390738 13365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:50.374677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41653 (local address 127.0.98.1:51233)
0619 16:51:50.375155 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:50.375203 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:50.375937 (+ 734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:50.379002 (+ 3065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:50.379008 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:50.379280 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:51:50.379640 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:50.379647 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:50.381241 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:50.381243 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:50.382508 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:50.382512 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:50.383367 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:50.383369 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:50.383534 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:50.384215 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:51:50.384227 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:51:50.386057 (+ 1830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:51:50.388488 (+ 2431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:50.388498 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:50.388501 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:50.388844 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:50.389172 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:50.389173 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:50.389173 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:50.389332 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:50.389875 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:51:50.389883 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:51:50.390081 (+ 198us) client_negotiation.cc:769] Sending connection context
0619 16:51:50.390217 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 16:51:50.390439 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":107,"threads_started":1}
I20260619 16:51:50.391788 13366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:50.374723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51233 (local address 127.0.98.62:41653)
0619 16:51:50.378415 (+ 3692us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:50.378430 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:50.378489 (+ 59us) server_negotiation.cc:407] Connection header received
0619 16:51:50.378666 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:50.378674 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:50.378739 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:50.379772 (+ 1033us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:50.379871 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:50.381117 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:50.382618 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:50.383244 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:50.383890 (+ 646us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:50.386215 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:50.386221 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:50.386222 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:50.386234 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:50.388369 (+ 2135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:50.388954 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:50.388958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:50.388958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:50.389089 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:50.389414 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:50.389415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:50.389415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:50.389781 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:50.391303 (+ 1522us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:50.391407 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 16:51:50.391563 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3593,"thread_start_us":84,"threads_started":1}
I20260619 16:51:50.393067 13363 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41653
I20260619 16:51:50.393198 13363 heartbeater.cc:464] Registering TS with master...
I20260619 16:51:50.393460 13363 heartbeater.cc:511] Master 127.0.98.62:41653 requested a full tablet report, sending...
W20260619 16:51:50.394467 13363 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41653 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 16:51:50.394550 13363 heartbeater.cc:464] Registering TS with master...
I20260619 16:51:50.394621 13363 heartbeater.cc:511] Master 127.0.98.62:41653 requested a full tablet report, sending...
I20260619 16:51:50.394934 13363 heartbeater.cc:464] Registering TS with master...
I20260619 16:51:50.395015 13363 heartbeater.cc:511] Master 127.0.98.62:41653 requested a full tablet report, sending...
W20260619 16:51:50.395318 13363 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 16:51:50.513875 13367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:51:50.514243 13367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:51:50.514293 13367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:51:50.516310 13367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:51:50.516399 13367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:51:50.516451 13367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:51:50.519109 13367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36039/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44409
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38507
--tserver_master_addrs=127.0.98.62:41653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:51:50.519472 13367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:51:50.519829 13367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:51:50.526050 13373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:51:50.526050 13372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:51:50.526069 13367 server_base.cc:1034] running on GCE node
W20260619 16:51:50.526342 13375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:51:50.526577 13367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:51:50.526937 13367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:51:50.528144 13367 hybrid_clock.cc:648] HybridClock initialized: now 1781887910528125 us; error 35 us; skew 500 ppm
Jun 19 16:51:50 dist-test-slave-jsw9 krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:51:50.531992 13367 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:51:50.532748 13367 webserver.cc:460] Webserver started at http://127.0.98.2:38507/ using document root <none> and password file <none>
I20260619 16:51:50.533133 13367 fs_manager.cc:359] Metadata directory not provided
I20260619 16:51:50.533288 13367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 16:51:50 dist-test-slave-jsw9 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:51:50.551565 13367 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 16:51:50.552603 13383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:50.552780 13367 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:51:50.552948 13367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-1/wal
uuid: "49734114abe9465b8382bf37e50a58c3"
format_stamp: "Formatted at 2026-06-19 16:51:49 on dist-test-slave-jsw9"
server_key: "9285e511203d04a096aac41f8dba4562"
server_key_iv: "2527e40486c676c2a693a2ba97e34eaa"
server_key_version: "kuduclusterkey@0"
I20260619 16:51:50.553537 13367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:51:50.571601 13367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:51:50.574239 13367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:51:50.574440 13367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:51:50.574764 13367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:51:50.575253 13367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:51:50.575309 13367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:50.575343 13367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:51:50.575361 13367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:50.586292 13367 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44409
I20260619 16:51:50.587208 13367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:51:50.587826 13496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44409 every 8 connection(s)
I20260619 16:51:50.589926 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 13367
I20260619 16:51:50.590274 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36039/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38927
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41653
--webserver_port=43915
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36039/kms with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:51:50 dist-test-slave-jsw9 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:51:50.604508 13366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:50.588686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43761 (local address 127.0.98.62:41653)
0619 16:51:50.588745 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:50.588748 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:50.588760 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:51:50.589529 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:50.589532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:50.589556 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:50.589591 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:50.590688 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:50.591518 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:50.593254 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:50.593567 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:50.593609 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:50.600421 (+ 6812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:50.600424 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:50.600424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:50.600437 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:50.602483 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:50.603159 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:50.603161 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:50.603161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:50.603238 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:50.603602 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:50.603603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:50.603603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:50.603774 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:50.603842 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:50.604278 (+ 436us) server_negotiation.cc:299] Negotiation successful
0619 16:51:50.604388 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 16:51:50.605121 13499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:50.588338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41653 (local address 127.0.98.2:43761)
0619 16:51:50.588575 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:50.588633 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:50.589379 (+ 746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:50.589868 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:50.589873 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:50.590175 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:51:50.590562 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:50.590571 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:50.591661 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:50.591662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:50.593112 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:50.593117 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:50.593720 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:50.593721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:50.593850 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:50.598078 (+ 4228us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:51:50.598096 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:51:50.600240 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:51:50.602612 (+ 2372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:50.602631 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:50.602637 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:50.603060 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:50.603320 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:50.603322 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:50.603322 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:50.603531 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:50.603853 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:51:50.603862 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:51:50.604123 (+ 261us) client_negotiation.cc:769] Sending connection context
0619 16:51:50.604259 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 16:51:50.604467 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":106,"threads_started":1}
I20260619 16:51:50.605943 13497 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41653
I20260619 16:51:50.606083 13497 heartbeater.cc:464] Registering TS with master...
I20260619 16:51:50.606382 13497 heartbeater.cc:511] Master 127.0.98.62:41653 requested a full tablet report, sending...
W20260619 16:51:50.607242 13497 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41653 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 16:51:50.607335 13497 heartbeater.cc:464] Registering TS with master...
I20260619 16:51:50.607411 13497 heartbeater.cc:511] Master 127.0.98.62:41653 requested a full tablet report, sending...
I20260619 16:51:50.607777 13497 heartbeater.cc:464] Registering TS with master...
I20260619 16:51:50.607874 13497 heartbeater.cc:511] Master 127.0.98.62:41653 requested a full tablet report, sending...
W20260619 16:51:50.608219 13497 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 16:51:50.719862 13500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:51:50.720184 13500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:51:50.720232 13500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:51:50.722482 13500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:51:50.722560 13500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:51:50.722611 13500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:51:50.725696 13500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:36039/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38927
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43915
--tserver_master_addrs=127.0.98.62:41653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:51:50.726094 13500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:51:50.726480 13500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:51:50.732779 13506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:51:50.732993 13509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:51:50.732779 13507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:51:50.733383 13500 server_base.cc:1034] running on GCE node
I20260619 16:51:50.733640 13500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:51:50.734002 13500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:51:50.735189 13500 hybrid_clock.cc:648] HybridClock initialized: now 1781887910735186 us; error 53 us; skew 500 ppm
Jun 19 16:51:50 dist-test-slave-jsw9 krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:51:50.738927 13500 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:51:50.739650 13500 webserver.cc:460] Webserver started at http://127.0.98.3:43915/ using document root <none> and password file <none>
I20260619 16:51:50.739967 13500 fs_manager.cc:359] Metadata directory not provided
I20260619 16:51:50.740060 13500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 16:51:50 dist-test-slave-jsw9 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:51:50.764931 13500 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:51:50.769665 13517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:50.769974 13500 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260619 16:51:50.770095 13500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-2/wal
uuid: "bf3f1e0cc2184097a0b0c8db60d890fc"
format_stamp: "Formatted at 2026-06-19 16:51:49 on dist-test-slave-jsw9"
server_key: "a98f3624bcaf0d3bbf12f5eca41011be"
server_key_iv: "0eebf98ff73595056b2ef91647af17e6"
server_key_version: "kuduclusterkey@0"
I20260619 16:51:50.770731 13500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:51:50.791752 13500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:51:50.794452 13500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:51:50.794631 13500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:51:50.794955 13500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:51:50.795369 13500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:51:50.795416 13500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:50.795440 13500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:51:50.795452 13500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:50.803306 13500 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38927
I20260619 16:51:50.803393 13630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38927 every 8 connection(s)
I20260619 16:51:50.803790 13500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:51:50.809491 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 13500
Jun 19 16:51:50 dist-test-slave-jsw9 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:51:50.815791 13366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:50.804664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43429 (local address 127.0.98.62:41653)
0619 16:51:50.804750 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:50.804753 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:50.805140 (+ 387us) server_negotiation.cc:407] Connection header received
0619 16:51:50.806147 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:50.806150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:50.806171 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:50.806277 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:50.807134 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:50.807685 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:50.809275 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:50.809564 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:50.809644 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:50.812483 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:50.812485 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:50.812486 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:50.812494 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:50.814143 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:50.814586 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:50.814587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:50.814588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:50.814651 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:50.814940 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:50.814941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:50.814941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:50.815107 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:50.815201 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:50.815594 (+ 393us) server_negotiation.cc:299] Negotiation successful
0619 16:51:50.815686 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 16:51:50.816064 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:50.804642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41653 (local address 127.0.98.3:43429)
0619 16:51:50.804910 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:50.804969 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:50.805996 (+ 1027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:50.806363 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:50.806369 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:50.806680 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:51:50.807008 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:50.807016 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:50.807794 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:50.807796 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:50.809154 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:50.809159 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:50.809754 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:50.809755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:50.809894 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:50.810604 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:51:50.810616 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:51:50.812380 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:51:50.814232 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:50.814241 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:50.814245 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:50.814517 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:50.814706 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:50.814707 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:50.814708 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:50.814866 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:50.815195 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:51:50.815202 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:51:50.815433 (+ 231us) client_negotiation.cc:769] Sending connection context
0619 16:51:50.815555 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 16:51:50.815733 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":92,"threads_started":1}
I20260619 16:51:50.816565 13631 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41653
I20260619 16:51:50.816704 13631 heartbeater.cc:464] Registering TS with master...
I20260619 16:51:50.816951 13631 heartbeater.cc:511] Master 127.0.98.62:41653 requested a full tablet report, sending...
W20260619 16:51:50.817713 13631 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41653 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 16:51:50.817793 13631 heartbeater.cc:464] Registering TS with master...
I20260619 16:51:50.817894 13631 heartbeater.cc:511] Master 127.0.98.62:41653 requested a full tablet report, sending...
I20260619 16:51:50.818265 13631 heartbeater.cc:464] Registering TS with master...
I20260619 16:51:50.818353 13631 heartbeater.cc:511] Master 127.0.98.62:41653 requested a full tablet report, sending...
W20260619 16:51:50.818648 13631 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 16:51:50.821055 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:50.810030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47904 (local address 127.0.98.62:41653)
0619 16:51:50.810287 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:50.810289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:50.810369 (+ 80us) server_negotiation.cc:407] Connection header received
0619 16:51:50.810462 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:50.810463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:50.810479 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:50.810567 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:50.812164 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:50.812961 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:50.814041 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:50.814698 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:50.814785 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:50.817474 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:50.817477 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:50.817477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:50.817485 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:50.819569 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:50.820043 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:50.820044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:50.820045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:50.820123 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:50.820398 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:50.820399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:50.820399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:50.820550 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:50.820769 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:50.820817 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 16:51:50.820921 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":83,"threads_started":1}
Jun 19 16:51:51 dist-test-slave-jsw9 krb5kdc[12168](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781887911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:51:51.395520 13363 heartbeater.cc:464] Registering TS with master...
I20260619 16:51:51.395716 13363 heartbeater.cc:511] Master 127.0.98.62:41653 requested a full tablet report, sending...
I20260619 16:51:51.396226 13165 ts_manager.cc:194] Registered new tserver with Master: 51c7f6f7290143f1939e915dfdd50453 (127.0.98.1:34241)
I20260619 16:51:51.397341 13165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51233
I20260619 16:51:51.608436 13497 heartbeater.cc:464] Registering TS with master...
I20260619 16:51:51.608661 13497 heartbeater.cc:511] Master 127.0.98.62:41653 requested a full tablet report, sending...
I20260619 16:51:51.609063 13165 ts_manager.cc:194] Registered new tserver with Master: 49734114abe9465b8382bf37e50a58c3 (127.0.98.2:44409)
I20260619 16:51:51.609498 13165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43761
Jun 19 16:51:51 dist-test-slave-jsw9 krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:51:51.818840 13631 heartbeater.cc:464] Registering TS with master...
I20260619 16:51:51.819053 13631 heartbeater.cc:511] Master 127.0.98.62:41653 requested a full tablet report, sending...
I20260619 16:51:51.819554 13165 ts_manager.cc:194] Registered new tserver with Master: bf3f1e0cc2184097a0b0c8db60d890fc (127.0.98.3:38927)
I20260619 16:51:51.819999 13165 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43429
I20260619 16:51:51.827907 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:51:51.838151 13652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:51.829126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47908 (local address 127.0.98.62:41653)
0619 16:51:51.829489 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:51.829491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:51.829675 (+ 184us) server_negotiation.cc:407] Connection header received
0619 16:51:51.829704 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:51.829704 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:51.829719 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:51.829778 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:51.830632 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:51.831515 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:51.832630 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:51.832948 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:51.833013 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:51.834652 (+ 1639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:51.834654 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:51.834654 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:51.834663 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:51.836493 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:51.836894 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:51.836896 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:51.836896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:51.836963 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:51.837152 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:51.837153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:51.837153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:51.837321 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:51.837419 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:51.837931 (+ 512us) server_negotiation.cc:299] Negotiation successful
0619 16:51:51.838030 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":88,"threads_started":1}
I20260619 16:51:51.841037 13165 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47908:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:51:51.844419 13165 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:51:51.869969 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:51.857660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38927 (local address 127.0.0.1:58590)
0619 16:51:51.862600 (+ 4940us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:51.862679 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:51.862787 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:51.863784 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:51.863788 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:51.863792 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:51:51.863904 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:51.863907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:51.866421 (+ 2514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:51.866423 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:51.867902 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:51.867906 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:51.869587 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:51.869587 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:51.869697 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:51.869715 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 16:51:51.869780 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 16:51:51.869848 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4894,"thread_start_us":75,"threads_started":1}
I20260619 16:51:51.870664 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:51.857788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58590 (local address 127.0.98.3:38927)
0619 16:51:51.862508 (+ 4720us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:51.862514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:51.863208 (+ 694us) server_negotiation.cc:407] Connection header received
0619 16:51:51.863375 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:51.863378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:51.863399 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:51.863491 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:51:51.864004 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:51.866318 (+ 2314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:51.868032 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:51.869021 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:51.869113 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:51.870469 (+ 1356us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:51.870510 (+ 41us) server_negotiation.cc:299] Negotiation successful
0619 16:51:51.870568 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4659,"thread_start_us":83,"threads_started":1}
I20260619 16:51:51.871943 13564 tablet_service.cc:1467] Processing CreateTablet for tablet 6960af45d1044e7d9c084a894a8fe8dd (DEFAULT_TABLE table=test-table [id=6b551aae801a4adbb244176179e31b8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:51:51.872429 13564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6960af45d1044e7d9c084a894a8fe8dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:51:51.872713 13663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:51.858081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44409 (local address 127.0.0.1:53568)
0619 16:51:51.866909 (+ 8828us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:51.866920 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:51.866977 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:51.867309 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:51.867310 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:51.867313 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:51:51.867370 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:51.867372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:51.870114 (+ 2742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:51.870115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:51.871343 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:51.871347 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:51.872448 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:51.872449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:51.872512 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:51.872517 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:51:51.872564 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 16:51:51.872613 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8808,"thread_start_us":73,"threads_started":1}
I20260619 16:51:51.873011 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:51.858472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53568 (local address 127.0.98.2:44409)
0619 16:51:51.862391 (+ 3919us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:51.862404 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:51.867046 (+ 4642us) server_negotiation.cc:407] Connection header received
0619 16:51:51.867094 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:51.867097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:51.867115 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:51.867187 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:51:51.867447 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:51.868559 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:51.871459 (+ 2900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:51.872255 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:51.872327 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:51.872409 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:51.872879 (+ 470us) server_negotiation.cc:299] Negotiation successful
0619 16:51:51.872928 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3787,"thread_start_us":73,"threads_started":1}
I20260619 16:51:51.873728 13429 tablet_service.cc:1467] Processing CreateTablet for tablet 6960af45d1044e7d9c084a894a8fe8dd (DEFAULT_TABLE table=test-table [id=6b551aae801a4adbb244176179e31b8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:51:51.874111 13429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6960af45d1044e7d9c084a894a8fe8dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:51:51.881644 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:51.857898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34241 (local address 127.0.0.1:36992)
0619 16:51:51.866437 (+ 8539us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:51.866451 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:51.866536 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:51.868670 (+ 2134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:51.868672 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:51.868675 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:51:51.868728 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:51.868730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:51.872318 (+ 3588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:51.872319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:51.879976 (+ 7657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:51.879981 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:51.881283 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:51.881284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:51.881369 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:51.881375 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 16:51:51.881429 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 16:51:51.881492 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8516,"thread_start_us":76,"threads_started":1}
I20260619 16:51:51.882474 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:51.858026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36992 (local address 127.0.98.1:34241)
0619 16:51:51.866354 (+ 8328us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:51.866360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:51.866601 (+ 241us) server_negotiation.cc:407] Connection header received
0619 16:51:51.866650 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:51.866653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:51.866672 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:51.866764 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:51:51.868819 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:51.870006 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:51.880099 (+ 10093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:51.881036 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:51.881123 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:51.881232 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:51.882139 (+ 907us) server_negotiation.cc:299] Negotiation successful
0619 16:51:51.882359 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8271,"thread_start_us":61,"threads_started":1}
I20260619 16:51:51.883865 13297 tablet_service.cc:1467] Processing CreateTablet for tablet 6960af45d1044e7d9c084a894a8fe8dd (DEFAULT_TABLE table=test-table [id=6b551aae801a4adbb244176179e31b8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:51:51.884373 13297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6960af45d1044e7d9c084a894a8fe8dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:51:51.887526 13665 tablet_bootstrap.cc:492] T 6960af45d1044e7d9c084a894a8fe8dd P bf3f1e0cc2184097a0b0c8db60d890fc: Bootstrap starting.
I20260619 16:51:51.888652 13665 tablet_bootstrap.cc:654] T 6960af45d1044e7d9c084a894a8fe8dd P bf3f1e0cc2184097a0b0c8db60d890fc: Neither blocks nor log segments found. Creating new log.
I20260619 16:51:51.889053 13665 log.cc:824] T 6960af45d1044e7d9c084a894a8fe8dd P bf3f1e0cc2184097a0b0c8db60d890fc: Log is configured to *not* fsync() on all Append() calls
I20260619 16:51:51.890106 13667 tablet_bootstrap.cc:492] T 6960af45d1044e7d9c084a894a8fe8dd P 51c7f6f7290143f1939e915dfdd50453: Bootstrap starting.
I20260619 16:51:51.890470 13666 tablet_bootstrap.cc:492] T 6960af45d1044e7d9c084a894a8fe8dd P 49734114abe9465b8382bf37e50a58c3: Bootstrap starting.
I20260619 16:51:51.891337 13667 tablet_bootstrap.cc:654] T 6960af45d1044e7d9c084a894a8fe8dd P 51c7f6f7290143f1939e915dfdd50453: Neither blocks nor log segments found. Creating new log.
I20260619 16:51:51.892251 13666 tablet_bootstrap.cc:654] T 6960af45d1044e7d9c084a894a8fe8dd P 49734114abe9465b8382bf37e50a58c3: Neither blocks nor log segments found. Creating new log.
I20260619 16:51:51.892663 13667 log.cc:824] T 6960af45d1044e7d9c084a894a8fe8dd P 51c7f6f7290143f1939e915dfdd50453: Log is configured to *not* fsync() on all Append() calls
I20260619 16:51:51.893087 13666 log.cc:824] T 6960af45d1044e7d9c084a894a8fe8dd P 49734114abe9465b8382bf37e50a58c3: Log is configured to *not* fsync() on all Append() calls
I20260619 16:51:51.894495 13667 tablet_bootstrap.cc:492] T 6960af45d1044e7d9c084a894a8fe8dd P 51c7f6f7290143f1939e915dfdd50453: No bootstrap required, opened a new log
I20260619 16:51:51.894620 13667 ts_tablet_manager.cc:1397] T 6960af45d1044e7d9c084a894a8fe8dd P 51c7f6f7290143f1939e915dfdd50453: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 16:51:51.898154 13667 raft_consensus.cc:348] T 6960af45d1044e7d9c084a894a8fe8dd P 51c7f6f7290143f1939e915dfdd50453 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3f1e0cc2184097a0b0c8db60d890fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38927 } } peers { permanent_uuid: "49734114abe9465b8382bf37e50a58c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44409 } } peers { permanent_uuid: "51c7f6f7290143f1939e915dfdd50453" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34241 } }
I20260619 16:51:51.898614 13667 raft_consensus.cc:374] T 6960af45d1044e7d9c084a894a8fe8dd P 51c7f6f7290143f1939e915dfdd50453 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:51:51.898698 13667 raft_consensus.cc:729] T 6960af45d1044e7d9c084a894a8fe8dd P 51c7f6f7290143f1939e915dfdd50453 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51c7f6f7290143f1939e915dfdd50453, State: Initialized, Role: FOLLOWER
I20260619 16:51:51.898928 13667 consensus_queue.cc:260] T 6960af45d1044e7d9c084a894a8fe8dd P 51c7f6f7290143f1939e915dfdd50453 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3f1e0cc2184097a0b0c8db60d890fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38927 } } peers { permanent_uuid: "49734114abe9465b8382bf37e50a58c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44409 } } peers { permanent_uuid: "51c7f6f7290143f1939e915dfdd50453" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34241 } }
I20260619 16:51:51.899343 13667 ts_tablet_manager.cc:1428] T 6960af45d1044e7d9c084a894a8fe8dd P 51c7f6f7290143f1939e915dfdd50453: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260619 16:51:51.899539 13363 heartbeater.cc:503] Master 127.0.98.62:41653 was elected leader, sending a full tablet report...
I20260619 16:51:51.901165 13666 tablet_bootstrap.cc:492] T 6960af45d1044e7d9c084a894a8fe8dd P 49734114abe9465b8382bf37e50a58c3: No bootstrap required, opened a new log
I20260619 16:51:51.901320 13666 ts_tablet_manager.cc:1397] T 6960af45d1044e7d9c084a894a8fe8dd P 49734114abe9465b8382bf37e50a58c3: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260619 16:51:51.901435 13665 tablet_bootstrap.cc:492] T 6960af45d1044e7d9c084a894a8fe8dd P bf3f1e0cc2184097a0b0c8db60d890fc: No bootstrap required, opened a new log
I20260619 16:51:51.901507 13665 ts_tablet_manager.cc:1397] T 6960af45d1044e7d9c084a894a8fe8dd P bf3f1e0cc2184097a0b0c8db60d890fc: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20260619 16:51:51.904466 13665 raft_consensus.cc:348] T 6960af45d1044e7d9c084a894a8fe8dd P bf3f1e0cc2184097a0b0c8db60d890fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3f1e0cc2184097a0b0c8db60d890fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38927 } } peers { permanent_uuid: "49734114abe9465b8382bf37e50a58c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44409 } } peers { permanent_uuid: "51c7f6f7290143f1939e915dfdd50453" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34241 } }
I20260619 16:51:51.904466 13666 raft_consensus.cc:348] T 6960af45d1044e7d9c084a894a8fe8dd P 49734114abe9465b8382bf37e50a58c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3f1e0cc2184097a0b0c8db60d890fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38927 } } peers { permanent_uuid: "49734114abe9465b8382bf37e50a58c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44409 } } peers { permanent_uuid: "51c7f6f7290143f1939e915dfdd50453" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34241 } }
I20260619 16:51:51.904696 13665 raft_consensus.cc:374] T 6960af45d1044e7d9c084a894a8fe8dd P bf3f1e0cc2184097a0b0c8db60d890fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:51:51.904737 13665 raft_consensus.cc:729] T 6960af45d1044e7d9c084a894a8fe8dd P bf3f1e0cc2184097a0b0c8db60d890fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf3f1e0cc2184097a0b0c8db60d890fc, State: Initialized, Role: FOLLOWER
I20260619 16:51:51.904832 13666 raft_consensus.cc:374] T 6960af45d1044e7d9c084a894a8fe8dd P 49734114abe9465b8382bf37e50a58c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:51:51.904917 13666 raft_consensus.cc:729] T 6960af45d1044e7d9c084a894a8fe8dd P 49734114abe9465b8382bf37e50a58c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49734114abe9465b8382bf37e50a58c3, State: Initialized, Role: FOLLOWER
I20260619 16:51:51.904908 13665 consensus_queue.cc:260] T 6960af45d1044e7d9c084a894a8fe8dd P bf3f1e0cc2184097a0b0c8db60d890fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3f1e0cc2184097a0b0c8db60d890fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38927 } } peers { permanent_uuid: "49734114abe9465b8382bf37e50a58c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44409 } } peers { permanent_uuid: "51c7f6f7290143f1939e915dfdd50453" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34241 } }
I20260619 16:51:51.905131 13666 consensus_queue.cc:260] T 6960af45d1044e7d9c084a894a8fe8dd P 49734114abe9465b8382bf37e50a58c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3f1e0cc2184097a0b0c8db60d890fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38927 } } peers { permanent_uuid: "49734114abe9465b8382bf37e50a58c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44409 } } peers { permanent_uuid: "51c7f6f7290143f1939e915dfdd50453" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34241 } }
I20260619 16:51:51.905295 13665 ts_tablet_manager.cc:1428] T 6960af45d1044e7d9c084a894a8fe8dd P bf3f1e0cc2184097a0b0c8db60d890fc: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 16:51:51.905459 13631 heartbeater.cc:503] Master 127.0.98.62:41653 was elected leader, sending a full tablet report...
I20260619 16:51:51.905490 13666 ts_tablet_manager.cc:1428] T 6960af45d1044e7d9c084a894a8fe8dd P 49734114abe9465b8382bf37e50a58c3: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 16:51:51.905678 13497 heartbeater.cc:503] Master 127.0.98.62:41653 was elected leader, sending a full tablet report...
I20260619 16:51:51.946300 13672 raft_consensus.cc:482] T 6960af45d1044e7d9c084a894a8fe8dd P bf3f1e0cc2184097a0b0c8db60d890fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:51:51.946516 13672 raft_consensus.cc:504] T 6960af45d1044e7d9c084a894a8fe8dd P bf3f1e0cc2184097a0b0c8db60d890fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3f1e0cc2184097a0b0c8db60d890fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38927 } } peers { permanent_uuid: "49734114abe9465b8382bf37e50a58c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44409 } } peers { permanent_uuid: "51c7f6f7290143f1939e915dfdd50453" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34241 } }
I20260619 16:51:51.947029 13672 leader_election.cc:288] T 6960af45d1044e7d9c084a894a8fe8dd P bf3f1e0cc2184097a0b0c8db60d890fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49734114abe9465b8382bf37e50a58c3 (127.0.98.2:44409), 51c7f6f7290143f1939e915dfdd50453 (127.0.98.1:34241)
I20260619 16:51:51.951680 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:51.947796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59061 (local address 127.0.98.1:34241)
0619 16:51:51.947856 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:51.947858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:51.947975 (+ 117us) server_negotiation.cc:407] Connection header received
0619 16:51:51.948073 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:51.948074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:51.948086 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:51.948112 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:51:51.948360 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:51.949093 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:51.950583 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:51.951238 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:51.951300 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:51.951317 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:51.951534 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 16:51:51.951590 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 16:51:51.951718 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:51.947384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55631 (local address 127.0.98.2:44409)
0619 16:51:51.947452 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:51.947454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:51.947596 (+ 142us) server_negotiation.cc:407] Connection header received
0619 16:51:51.947709 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:51.947710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:51.947725 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:51.947757 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:51:51.948029 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:51.948862 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:51.950645 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:51.951304 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:51.951358 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:51.951370 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:51.951592 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 16:51:51.951645 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 16:51:51.952021 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:51.947292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44409 (local address 127.0.98.3:55631)
0619 16:51:51.947498 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:51.947511 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:51.947629 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:51.947833 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:51.947834 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:51.947836 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:51:51.947954 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:51.947957 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:51.948990 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:51.948991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:51.950510 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:51.950513 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:51.951445 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:51.951446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:51.951520 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:51.951524 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:51:51.951844 (+ 320us) client_negotiation.cc:240] Negotiation successful
0619 16:51:51.951900 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":139,"threads_started":1}
I20260619 16:51:51.952026 13675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:51.947582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34241 (local address 127.0.98.3:59061)
0619 16:51:51.947910 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:51.947920 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:51.947999 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:51.948177 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:51.948178 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:51.948179 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:51:51.948255 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:51.948258 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:51.949198 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:51.949199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:51.950459 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:51.950463 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:51.951384 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:51.951385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:51.951458 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:51.951463 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:51:51.951853 (+ 390us) client_negotiation.cc:240] Negotiation successful
0619 16:51:51.951907 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":82,"threads_started":1}
I20260619 16:51:51.952435 13439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6960af45d1044e7d9c084a894a8fe8dd" candidate_uuid: "bf3f1e0cc2184097a0b0c8db60d890fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49734114abe9465b8382bf37e50a58c3" is_pre_election: true
I20260619 16:51:51.952430 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6960af45d1044e7d9c084a894a8fe8dd" candidate_uuid: "bf3f1e0cc2184097a0b0c8db60d890fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51c7f6f7290143f1939e915dfdd50453" is_pre_election: true
I20260619 16:51:51.952608 13439 raft_consensus.cc:2454] T 6960af45d1044e7d9c084a894a8fe8dd P 49734114abe9465b8382bf37e50a58c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf3f1e0cc2184097a0b0c8db60d890fc in term 0.
I20260619 16:51:51.952610 13314 raft_consensus.cc:2454] T 6960af45d1044e7d9c084a894a8fe8dd P 51c7f6f7290143f1939e915dfdd50453 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf3f1e0cc2184097a0b0c8db60d890fc in term 0.
I20260619 16:51:51.952889 13521 leader_election.cc:302] T 6960af45d1044e7d9c084a894a8fe8dd P bf3f1e0cc2184097a0b0c8db60d890fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51c7f6f7290143f1939e915dfdd50453, bf3f1e0cc2184097a0b0c8db60d890fc; no voters:
I20260619 16:51:51.953060 13672 raft_consensus.cc:2781] T 6960af45d1044e7d9c084a894a8fe8dd P bf3f1e0cc2184097a0b0c8db60d890fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:51:51.953122 13672 raft_consensus.cc:482] T 6960af45d1044e7d9c084a894a8fe8dd P bf3f1e0cc2184097a0b0c8db60d890fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:51:51.953159 13672 raft_consensus.cc:3037] T 6960af45d1044e7d9c084a894a8fe8dd P bf3f1e0cc2184097a0b0c8db60d890fc [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:51:51.954350 13672 raft_consensus.cc:504] T 6960af45d1044e7d9c084a894a8fe8dd P bf3f1e0cc2184097a0b0c8db60d890fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3f1e0cc2184097a0b0c8db60d890fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38927 } } peers { permanent_uuid: "49734114abe9465b8382bf37e50a58c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44409 } } peers { permanent_uuid: "51c7f6f7290143f1939e915dfdd50453" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34241 } }
I20260619 16:51:51.954524 13672 leader_election.cc:288] T 6960af45d1044e7d9c084a894a8fe8dd P bf3f1e0cc2184097a0b0c8db60d890fc [CANDIDATE]: Term 1 election: Requested vote from peers 49734114abe9465b8382bf37e50a58c3 (127.0.98.2:44409), 51c7f6f7290143f1939e915dfdd50453 (127.0.98.1:34241)
I20260619 16:51:51.954708 13439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6960af45d1044e7d9c084a894a8fe8dd" candidate_uuid: "bf3f1e0cc2184097a0b0c8db60d890fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49734114abe9465b8382bf37e50a58c3"
I20260619 16:51:51.954794 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6960af45d1044e7d9c084a894a8fe8dd" candidate_uuid: "bf3f1e0cc2184097a0b0c8db60d890fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51c7f6f7290143f1939e915dfdd50453"
I20260619 16:51:51.954828 13439 raft_consensus.cc:3037] T 6960af45d1044e7d9c084a894a8fe8dd P 49734114abe9465b8382bf37e50a58c3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:51:51.954886 13314 raft_consensus.cc:3037] T 6960af45d1044e7d9c084a894a8fe8dd P 51c7f6f7290143f1939e915dfdd50453 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:51:51.955920 13314 raft_consensus.cc:2454] T 6960af45d1044e7d9c084a894a8fe8dd P 51c7f6f7290143f1939e915dfdd50453 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf3f1e0cc2184097a0b0c8db60d890fc in term 1.
I20260619 16:51:51.955925 13439 raft_consensus.cc:2454] T 6960af45d1044e7d9c084a894a8fe8dd P 49734114abe9465b8382bf37e50a58c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf3f1e0cc2184097a0b0c8db60d890fc in term 1.
I20260619 16:51:51.956172 13518 leader_election.cc:302] T 6960af45d1044e7d9c084a894a8fe8dd P bf3f1e0cc2184097a0b0c8db60d890fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49734114abe9465b8382bf37e50a58c3, bf3f1e0cc2184097a0b0c8db60d890fc; no voters:
I20260619 16:51:51.956449 13672 raft_consensus.cc:2781] T 6960af45d1044e7d9c084a894a8fe8dd P bf3f1e0cc2184097a0b0c8db60d890fc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:51:51.956671 13672 raft_consensus.cc:686] T 6960af45d1044e7d9c084a894a8fe8dd P bf3f1e0cc2184097a0b0c8db60d890fc [term 1 LEADER]: Becoming Leader. State: Replica: bf3f1e0cc2184097a0b0c8db60d890fc, State: Running, Role: LEADER
I20260619 16:51:51.956792 13672 consensus_queue.cc:237] T 6960af45d1044e7d9c084a894a8fe8dd P bf3f1e0cc2184097a0b0c8db60d890fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3f1e0cc2184097a0b0c8db60d890fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38927 } } peers { permanent_uuid: "49734114abe9465b8382bf37e50a58c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44409 } } peers { permanent_uuid: "51c7f6f7290143f1939e915dfdd50453" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34241 } }
I20260619 16:51:51.957809 13164 catalog_manager.cc:5495] T 6960af45d1044e7d9c084a894a8fe8dd P bf3f1e0cc2184097a0b0c8db60d890fc reported cstate change: term changed from 0 to 1, leader changed from <none> to bf3f1e0cc2184097a0b0c8db60d890fc (127.0.98.3). New cstate: current_term: 1 leader_uuid: "bf3f1e0cc2184097a0b0c8db60d890fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3f1e0cc2184097a0b0c8db60d890fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49734114abe9465b8382bf37e50a58c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51c7f6f7290143f1939e915dfdd50453" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34241 } health_report { overall_health: UNKNOWN } } }
I20260619 16:51:51.984004 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:51.980620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58592 (local address 127.0.98.3:38927)
0619 16:51:51.980689 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:51.980691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:51.980700 (+ 9us) server_negotiation.cc:407] Connection header received
0619 16:51:51.980750 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:51.980751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:51.980781 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:51.980864 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:51:51.981058 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:51.981819 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:51.983012 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:51.983321 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:51.983392 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:51.983581 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:51:51.983754 (+ 173us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:51:51.983849 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:51.983875 (+ 26us) server_negotiation.cc:299] Negotiation successful
0619 16:51:51.983906 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 16:51:51.986716 13314 raft_consensus.cc:1261] T 6960af45d1044e7d9c084a894a8fe8dd P 51c7f6f7290143f1939e915dfdd50453 [term 1 FOLLOWER]: Refusing update from remote peer bf3f1e0cc2184097a0b0c8db60d890fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:51:51.986747 13439 raft_consensus.cc:1261] T 6960af45d1044e7d9c084a894a8fe8dd P 49734114abe9465b8382bf37e50a58c3 [term 1 FOLLOWER]: Refusing update from remote peer bf3f1e0cc2184097a0b0c8db60d890fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:51:51.987112 13672 consensus_queue.cc:1035] T 6960af45d1044e7d9c084a894a8fe8dd P bf3f1e0cc2184097a0b0c8db60d890fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "51c7f6f7290143f1939e915dfdd50453" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:51:51.987358 13676 consensus_queue.cc:1035] T 6960af45d1044e7d9c084a894a8fe8dd P bf3f1e0cc2184097a0b0c8db60d890fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "49734114abe9465b8382bf37e50a58c3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:51:51.992231 13680 mvcc.cc:205] Tried to move back new op lower bound from 7298612887494012928 to 7298612887375810560. Current Snapshot: MvccSnapshot[applied={T|T < 7298612887494012928}]
I20260619 16:51:51.992885 13679 mvcc.cc:205] Tried to move back new op lower bound from 7298612887494012928 to 7298612887375810560. Current Snapshot: MvccSnapshot[applied={T|T < 7298612887494012928}]
I20260619 16:51:51.993566 13681 mvcc.cc:205] Tried to move back new op lower bound from 7298612887494012928 to 7298612887375810560. Current Snapshot: MvccSnapshot[applied={T|T < 7298612887494012928}]
I20260619 16:51:51.997180 13165 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:51:51.997329 13165 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:51:51.998533 13165 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7b09c65a3cda4ed380f2ba7e5dd6e2be: Sending DeleteTablet for 3 replicas of tablet 6960af45d1044e7d9c084a894a8fe8dd
I20260619 16:51:51.999020 13297 tablet_service.cc:1514] Processing DeleteTablet for tablet 6960af45d1044e7d9c084a894a8fe8dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:51:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36992
I20260619 16:51:51.999608 13429 tablet_service.cc:1514] Processing DeleteTablet for tablet 6960af45d1044e7d9c084a894a8fe8dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:51:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53568
I20260619 16:51:52.000085 13564 tablet_service.cc:1514] Processing DeleteTablet for tablet 6960af45d1044e7d9c084a894a8fe8dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:51:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58590
I20260619 16:51:52.000746 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 13203
I20260619 16:51:52.002059 13689 tablet_replica.cc:330] T 6960af45d1044e7d9c084a894a8fe8dd P 49734114abe9465b8382bf37e50a58c3: stopping tablet replica
I20260619 16:51:52.002296 13689 raft_consensus.cc:2229] T 6960af45d1044e7d9c084a894a8fe8dd P 49734114abe9465b8382bf37e50a58c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:51:52.002533 13689 raft_consensus.cc:2258] T 6960af45d1044e7d9c084a894a8fe8dd P 49734114abe9465b8382bf37e50a58c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:51:52.002732 13690 tablet_replica.cc:330] T 6960af45d1044e7d9c084a894a8fe8dd P bf3f1e0cc2184097a0b0c8db60d890fc: stopping tablet replica
I20260619 16:51:52.003033 13690 raft_consensus.cc:2229] T 6960af45d1044e7d9c084a894a8fe8dd P bf3f1e0cc2184097a0b0c8db60d890fc [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:51:52.003232 13690 raft_consensus.cc:2258] T 6960af45d1044e7d9c084a894a8fe8dd P bf3f1e0cc2184097a0b0c8db60d890fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:51:52.003814 13690 ts_tablet_manager.cc:1905] T 6960af45d1044e7d9c084a894a8fe8dd P bf3f1e0cc2184097a0b0c8db60d890fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:51:52.006886 13690 ts_tablet_manager.cc:1918] T 6960af45d1044e7d9c084a894a8fe8dd P bf3f1e0cc2184097a0b0c8db60d890fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:51:52.007150 13690 log.cc:1196] T 6960af45d1044e7d9c084a894a8fe8dd P bf3f1e0cc2184097a0b0c8db60d890fc: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781887741903642-392-0/minicluster-data/ts-2/wal/wals/6960af45d1044e7d9c084a894a8fe8dd
I20260619 16:51:52.007612 13690 ts_tablet_manager.cc:1939] T 6960af45d1044e7d9c084a894a8fe8dd P bf3f1e0cc2184097a0b0c8db60d890fc: Deleting consensus metadata
I20260619 16:51:52.008337 13152 catalog_manager.cc:4841] TS bf3f1e0cc2184097a0b0c8db60d890fc (127.0.98.3:38927): tablet 6960af45d1044e7d9c084a894a8fe8dd (table test-table [id=6b551aae801a4adbb244176179e31b8b]) successfully deleted
W20260619 16:51:52.008946 13152 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34241 (error 108)
W20260619 16:51:52.009640 13661 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:51:52.009478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34241 (local address 127.0.0.1:37002)
0619 16:51:52.009539 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:52.009586 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34241: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 16:51:52.009761 13152 catalog_manager.cc:4584] TS 51c7f6f7290143f1939e915dfdd50453 (127.0.98.1:34241): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6960af45d1044e7d9c084a894a8fe8dd: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34241: connect: Connection reset by peer (error 104)
I20260619 16:51:52.010140 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 13367
W20260619 16:51:52.017503 13661 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:51:52.017100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44409 (local address 127.0.0.1:53572)
0619 16:51:52.017157 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:52.017169 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:52.017211 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:52.017443 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44409: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 16:51:52.018303 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 13500
I20260619 16:51:52.025890 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 13132
I20260619 16:51:52.031754 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 16:51:52.367069 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 16:51:52.367190 392 mini_ranger.cc:66] Stopping Ranger...
I20260619 16:51:52.713152 392 mini_ranger.cc:68] Stopped Ranger
2026-06-19 16:51:52.713 UTC [12198] LOG: received smart shutdown request
2026-06-19 16:51:52.717 UTC [12198] LOG: background worker "logical replication launcher" (PID 12207) exited with exit code 1
2026-06-19 16:51:52.717 UTC [12202] LOG: shutting down
2026-06-19 16:51:52.877 UTC [12198] LOG: database system is shut down
2026-06-19T16:51:52Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (55705 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:51:53 dist-test-slave-jsw9 krb5kdc[13700](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:51:53 dist-test-slave-jsw9 krb5kdc[13700](info): set up 2 sockets
Jun 19 16:51:53 dist-test-slave-jsw9 krb5kdc[13700](info): commencing operation
krb5kdc: starting...
W20260619 16:51:53.954548 392 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:51:54 dist-test-slave-jsw9 krb5kdc[13700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:51:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:51:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:51:54.087507 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36589
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36589,127.0.98.61:36603,127.0.98.60:38073
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:51:54.184155 13713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:51:54.184444 13713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:51:54.184473 13713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:51:54.186048 13713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:51:54.186126 13713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:51:54.186142 13713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:51:54.186154 13713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:51:54.186167 13713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:51:54.188094 13713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36589,127.0.98.61:36603,127.0.98.60:38073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:51:54.188398 13713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:51:54.188684 13713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:51:54.195322 13718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:51:54.195533 13713 server_base.cc:1034] running on GCE node
W20260619 16:51:54.195333 13719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:51:54.195415 13721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:51:54.196194 13713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:51:54.196564 13713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:51:54.197744 13713 hybrid_clock.cc:648] HybridClock initialized: now 1781887914197699 us; error 62 us; skew 500 ppm
Jun 19 16:51:54 dist-test-slave-jsw9 krb5kdc[13700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:51:54.200870 13713 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:51:54.201428 13713 webserver.cc:460] Webserver started at http://127.0.98.62:34693/ using document root <none> and password file <none>
I20260619 16:51:54.201673 13713 fs_manager.cc:359] Metadata directory not provided
I20260619 16:51:54.201725 13713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:51:54.201864 13713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:51:54.203163 13713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "d6737f0a071645768c81f32251f69c4a"
format_stamp: "Formatted at 2026-06-19 16:51:54 on dist-test-slave-jsw9"
server_key: "0341dd26ca4d25aaeb7960bb3260a0c8"
server_key_iv: "1e5b4e3c0461b0dfc12111ebbe2a9709"
server_key_version: "encryptionkey@0"
I20260619 16:51:54.203562 13713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "d6737f0a071645768c81f32251f69c4a"
format_stamp: "Formatted at 2026-06-19 16:51:54 on dist-test-slave-jsw9"
server_key: "0341dd26ca4d25aaeb7960bb3260a0c8"
server_key_iv: "1e5b4e3c0461b0dfc12111ebbe2a9709"
server_key_version: "encryptionkey@0"
I20260619 16:51:54.205755 13713 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:51:54.207175 13728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:54.207410 13713 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:51:54.207513 13713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "d6737f0a071645768c81f32251f69c4a"
format_stamp: "Formatted at 2026-06-19 16:51:54 on dist-test-slave-jsw9"
server_key: "0341dd26ca4d25aaeb7960bb3260a0c8"
server_key_iv: "1e5b4e3c0461b0dfc12111ebbe2a9709"
server_key_version: "encryptionkey@0"
I20260619 16:51:54.207589 13713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:51:54.226941 13713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:51:54.229605 13713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:51:54.229745 13713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:51:54.234828 13713 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36589
I20260619 16:51:54.234870 13780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36589 every 8 connection(s)
I20260619 16:51:54.235342 13713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:51:54.236344 13781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:51:54.238240 13781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36589 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:51:54.244297 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 13713
I20260619 16:51:54.244437 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:51:54.244627 392 external_mini_cluster.cc:1428] Setting key 296bf70ce0670f80c1534a91184a8ae2
Jun 19 16:51:54 dist-test-slave-jsw9 krb5kdc[13700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:51:54.252441 13784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.239573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37808 (local address 127.0.98.62:36589)
0619 16:51:54.239880 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:54.239885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:54.240059 (+ 174us) server_negotiation.cc:407] Connection header received
0619 16:51:54.240875 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:54.240878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:54.240905 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:54.241038 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:54.241791 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.242927 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.244097 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.244637 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.244792 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:54.248058 (+ 3266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:54.248067 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:54.248068 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:54.248076 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:54.249992 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:54.250600 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:54.250604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:54.250605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:54.250804 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:54.251175 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:54.251178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:54.251178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:54.251662 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:54.251741 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:54.252058 (+ 317us) server_negotiation.cc:299] Negotiation successful
0619 16:51:54.252194 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":183,"threads_started":1}
I20260619 16:51:54.252446 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.239410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36589 (local address 127.0.0.1:37808)
0619 16:51:54.239761 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:54.239835 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:54.240652 (+ 817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:54.241101 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:54.241105 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:54.241448 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:51:54.241666 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:54.241671 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.243050 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.243051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:54.243889 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:54.243892 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.244816 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.244817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:54.244939 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:54.245623 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:51:54.245636 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:51:54.247917 (+ 2281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:51:54.250098 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:54.250108 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:54.250112 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:54.250476 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:54.250882 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:54.250883 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:54.250883 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:54.251007 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:54.251755 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:51:54.251765 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:51:54.251872 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 16:51:54.252010 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 16:51:54.252290 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":80,"thread_start_us":127,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260619 16:51:54.254211 13781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36603 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:51:54.254791 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.254585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36603 (local address 127.0.0.1:33336)
0619 16:51:54.254684 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:54.254754 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260619 16:51:54.255049 13732 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36603: connect: Connection refused (error 111)
I20260619 16:51:54.255723 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.255597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36603 (local address 127.0.0.1:33346)
0619 16:51:54.255659 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:54.255692 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 16:51:54.255923 13781 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36603: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36603: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 16:51:54.274415 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36603
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36589,127.0.98.61:36603,127.0.98.60:38073
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:51:54.305253 13781 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36603 } attempt: 1
I20260619 16:51:54.305989 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.305794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36603 (local address 127.0.0.1:33358)
0619 16:51:54.305878 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:54.305943 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 16:51:54.306597 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.306449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36603 (local address 127.0.0.1:33374)
0619 16:51:54.306506 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:54.306572 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 16:51:54.306763 13781 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36603: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36603: connect: Connection refused (error 111)
I20260619 16:51:54.374985 13781 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36603 } attempt: 2
W20260619 16:51:54.375103 13788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:51:54.375391 13788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:51:54.375437 13788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 16:51:54.375571 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.375421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36603 (local address 127.0.0.1:33382)
0619 16:51:54.375480 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:54.375534 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 16:51:54.375901 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.375818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36603 (local address 127.0.0.1:33394)
0619 16:51:54.375855 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:54.375880 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 16:51:54.376000 13781 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36603: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36603: connect: Connection refused (error 111)
W20260619 16:51:54.376961 13788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:51:54.377010 13788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:51:54.377030 13788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:51:54.377048 13788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:51:54.377071 13788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:51:54.378898 13788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36589,127.0.98.61:36603,127.0.98.60:38073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:51:54.379185 13788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:51:54.379447 13788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:51:54.384927 13793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:51:54.384910 13794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:51:54.384904 13796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:51:54.384946 13788 server_base.cc:1034] running on GCE node
I20260619 16:51:54.385361 13788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:51:54.385696 13788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:51:54.386864 13788 hybrid_clock.cc:648] HybridClock initialized: now 1781887914386850 us; error 35 us; skew 500 ppm
Jun 19 16:51:54 dist-test-slave-jsw9 krb5kdc[13700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:51:54.389719 13788 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 16:51:54.390260 13788 webserver.cc:460] Webserver started at http://127.0.98.61:39925/ using document root <none> and password file <none>
I20260619 16:51:54.390487 13788 fs_manager.cc:359] Metadata directory not provided
I20260619 16:51:54.390538 13788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:51:54.390676 13788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:51:54.392130 13788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-1/data/instance:
uuid: "dc8a602d13d14e718be4332560308f5f"
format_stamp: "Formatted at 2026-06-19 16:51:54 on dist-test-slave-jsw9"
server_key: "64c4adc263df051dad18cbdf6b564a05"
server_key_iv: "17901cf40a02c22592396935ba65ea1a"
server_key_version: "encryptionkey@0"
I20260619 16:51:54.392766 13788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-1/wal/instance:
uuid: "dc8a602d13d14e718be4332560308f5f"
format_stamp: "Formatted at 2026-06-19 16:51:54 on dist-test-slave-jsw9"
server_key: "64c4adc263df051dad18cbdf6b564a05"
server_key_iv: "17901cf40a02c22592396935ba65ea1a"
server_key_version: "encryptionkey@0"
I20260619 16:51:54.394271 13788 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260619 16:51:54.395128 13803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:54.395287 13788 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:51:54.395381 13788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-1/wal
uuid: "dc8a602d13d14e718be4332560308f5f"
format_stamp: "Formatted at 2026-06-19 16:51:54 on dist-test-slave-jsw9"
server_key: "64c4adc263df051dad18cbdf6b564a05"
server_key_iv: "17901cf40a02c22592396935ba65ea1a"
server_key_version: "encryptionkey@0"
I20260619 16:51:54.395445 13788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:51:54.417748 13788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:51:54.419905 13788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:51:54.420058 13788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:51:54.424686 13788 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36603
I20260619 16:51:54.424731 13855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36603 every 8 connection(s)
I20260619 16:51:54.425127 13788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-1/data/info.pb
I20260619 16:51:54.426229 13856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:51:54.428056 13856 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36589 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:51:54.430734 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 13788
I20260619 16:51:54.430857 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-1/wal/instance
I20260619 16:51:54.431041 392 external_mini_cluster.cc:1428] Setting key 4eee87e849f52f378732e1f5417c602f
Jun 19 16:51:54 dist-test-slave-jsw9 krb5kdc[13700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:51:54.438812 13784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.428905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37824 (local address 127.0.98.62:36589)
0619 16:51:54.428961 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:54.428963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:54.429178 (+ 215us) server_negotiation.cc:407] Connection header received
0619 16:51:54.429708 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:54.429709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:54.429721 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:54.429775 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:54.430529 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.431147 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.432610 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.432936 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.433020 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:54.435430 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:54.435432 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:54.435432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:54.435441 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:54.436990 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:54.437573 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:54.437574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:54.437574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:54.437642 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:54.437962 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:54.437963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:54.437964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:54.438100 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:54.438171 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:54.438607 (+ 436us) server_negotiation.cc:299] Negotiation successful
0619 16:51:54.438700 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 16:51:54.439054 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.428868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36589 (local address 127.0.0.1:37824)
0619 16:51:54.429056 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:54.429100 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:54.429617 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:54.429840 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:54.429844 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:54.430136 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:51:54.430419 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:54.430424 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.431272 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.431273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:54.432492 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:54.432496 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.433036 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.433037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:54.433176 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:54.433787 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:51:54.433798 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:51:54.435327 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:51:54.437080 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:54.437091 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:54.437095 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:54.437466 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:54.437715 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:54.437716 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:54.437716 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:54.437877 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:54.438170 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:51:54.438178 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:51:54.438420 (+ 242us) client_negotiation.cc:769] Sending connection context
0619 16:51:54.438561 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 16:51:54.438771 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260619 16:51:54.439718 13856 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36603 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 19 16:51:54 dist-test-slave-jsw9 krb5kdc[13700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 16:51:54.447825 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.440126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33396 (local address 127.0.98.61:36603)
0619 16:51:54.440299 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:54.440304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:54.440321 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:51:54.440339 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:54.440340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:54.440353 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:54.440420 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:54.440737 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.441467 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.442212 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.442626 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.442694 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:54.444846 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:54.444857 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:54.444857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:54.444864 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:54.446591 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:54.446994 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:54.446998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:54.446998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:54.447052 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:54.447211 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:54.447212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:54.447213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:54.447385 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:54.447461 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:54.447586 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 16:51:54.447690 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":142,"thread_start_us":129,"threads_started":1}
I20260619 16:51:54.447896 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.440053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36603 (local address 127.0.0.1:33396)
0619 16:51:54.440104 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:54.440115 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:54.440156 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:54.440431 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:54.440432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:54.440575 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:51:54.440639 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:54.440641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.441542 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.441543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:54.442113 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:54.442115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.442724 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.442725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:54.442831 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:54.443309 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:51:54.443314 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:51:54.444738 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:51:54.446688 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:54.446689 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:54.446689 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:54.446912 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:54.447105 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:54.447106 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:54.447106 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:54.447161 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:54.447444 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:51:54.447445 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:51:54.447485 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 16:51:54.447553 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 16:51:54.447784 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":89}
I20260619 16:51:54.448560 13856 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38073 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:51:54.448983 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.448861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38073 (local address 127.0.0.1:48452)
0619 16:51:54.448906 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:54.448959 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260619 16:51:54.449100 13805 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38073: connect: Connection refused (error 111)
I20260619 16:51:54.449539 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.449451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38073 (local address 127.0.0.1:48464)
0619 16:51:54.449489 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:54.449515 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 16:51:54.449640 13856 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38073: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38073: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 16:51:54.462500 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38073
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36589,127.0.98.61:36603,127.0.98.60:38073
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:51:54.467161 13781 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36603 } attempt: 3
Jun 19 16:51:54 dist-test-slave-jsw9 krb5kdc[13700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 16:51:54.474396 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.467672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33404 (local address 127.0.98.61:36603)
0619 16:51:54.467746 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:54.467748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:54.467759 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:51:54.467791 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:54.467792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:54.467828 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:54.467874 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:54.468215 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.468781 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.469490 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.469707 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.469765 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:54.471936 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:54.471938 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:54.471938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:54.471948 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:54.473315 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:54.473632 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:54.473633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:54.473634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:54.473687 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:54.473847 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:54.473848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:54.473848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:54.473971 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:54.474029 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:54.474167 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 16:51:54.474285 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 16:51:54.474421 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.467585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36603 (local address 127.0.0.1:33404)
0619 16:51:54.467657 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:54.467669 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:54.467716 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:54.467887 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:54.467888 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:54.468061 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:51:54.468135 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:54.468137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.468867 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.468868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:54.469407 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:54.469408 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.469768 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.469769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:54.469847 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:54.470330 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:51:54.470335 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:51:54.471844 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:51:54.473398 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:54.473399 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:54.473400 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:54.473570 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:54.473740 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:54.473741 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:54.473741 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:54.473800 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:54.474025 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:51:54.474026 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:51:54.474078 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 16:51:54.474138 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 16:51:54.474331 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 16:51:54.475087 13781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38073 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:51:54.475471 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.475376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38073 (local address 127.0.0.1:48466)
0619 16:51:54.475421 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:54.475452 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 16:51:54.475766 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.475685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38073 (local address 127.0.0.1:48480)
0619 16:51:54.475723 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:54.475746 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 16:51:54.475871 13781 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38073: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38073: connect: Connection refused (error 111)
I20260619 16:51:54.498890 13856 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38073 } attempt: 1
I20260619 16:51:54.499502 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.499306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38073 (local address 127.0.0.1:48488)
0619 16:51:54.499396 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:54.499454 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 16:51:54.499918 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.499840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38073 (local address 127.0.0.1:48496)
0619 16:51:54.499869 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:54.499898 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260619 16:51:54.500043 13856 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38073: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38073: connect: Connection refused (error 111)
I20260619 16:51:54.507014 13781 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38073 } attempt: 1
I20260619 16:51:54.507627 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.507454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38073 (local address 127.0.0.1:48510)
0619 16:51:54.507528 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:54.507583 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 16:51:54.507990 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.507896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38073 (local address 127.0.0.1:48512)
0619 16:51:54.507939 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:54.507970 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 16:51:54.508082 13781 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38073: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38073: connect: Connection refused (error 111)
W20260619 16:51:54.557473 13863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:51:54.557932 13863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:51:54.557963 13863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:51:54.559455 13863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:51:54.559510 13863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:51:54.559523 13863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:51:54.559535 13863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:51:54.559546 13863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:51:54.561326 13863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36589,127.0.98.61:36603,127.0.98.60:38073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:51:54.561638 13863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:51:54.561918 13863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:51:54.567481 13868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:51:54.567446 13869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:51:54.567629 13863 server_base.cc:1034] running on GCE node
W20260619 16:51:54.567720 13871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:51:54.568082 13863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:51:54.568241 13856 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38073 } attempt: 2
I20260619 16:51:54.568531 13863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:51:54.568794 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.568647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38073 (local address 127.0.0.1:48518)
0619 16:51:54.568710 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:54.568750 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 16:51:54.569165 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.569043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38073 (local address 127.0.0.1:48522)
0619 16:51:54.569099 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:54.569128 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 16:51:54.569248 13856 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38073: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38073: connect: Connection refused (error 111)
I20260619 16:51:54.569718 13863 hybrid_clock.cc:648] HybridClock initialized: now 1781887914569701 us; error 34 us; skew 500 ppm
Jun 19 16:51:54 dist-test-slave-jsw9 krb5kdc[13700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:51:54.572458 13863 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 16:51:54.572934 13863 webserver.cc:460] Webserver started at http://127.0.98.60:45571/ using document root <none> and password file <none>
I20260619 16:51:54.573148 13863 fs_manager.cc:359] Metadata directory not provided
I20260619 16:51:54.573190 13863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:51:54.573297 13863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:51:54.574555 13863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-2/data/instance:
uuid: "2dabd470e02944a5a667495276b1a053"
format_stamp: "Formatted at 2026-06-19 16:51:54 on dist-test-slave-jsw9"
server_key: "3ed5125cd45b5f2fd68822654c212ae0"
server_key_iv: "3929c30320a73d77a32d4b4aaa305be5"
server_key_version: "encryptionkey@0"
I20260619 16:51:54.574896 13863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-2/wal/instance:
uuid: "2dabd470e02944a5a667495276b1a053"
format_stamp: "Formatted at 2026-06-19 16:51:54 on dist-test-slave-jsw9"
server_key: "3ed5125cd45b5f2fd68822654c212ae0"
server_key_iv: "3929c30320a73d77a32d4b4aaa305be5"
server_key_version: "encryptionkey@0"
I20260619 16:51:54.576308 13863 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 16:51:54.577195 13878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:54.577383 13863 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:51:54.577473 13863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-2/wal
uuid: "2dabd470e02944a5a667495276b1a053"
format_stamp: "Formatted at 2026-06-19 16:51:54 on dist-test-slave-jsw9"
server_key: "3ed5125cd45b5f2fd68822654c212ae0"
server_key_iv: "3929c30320a73d77a32d4b4aaa305be5"
server_key_version: "encryptionkey@0"
I20260619 16:51:54.577634 13863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:51:54.583241 13781 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38073 } attempt: 2
I20260619 16:51:54.583791 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.583659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38073 (local address 127.0.0.1:48534)
0619 16:51:54.583716 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:54.583756 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 16:51:54.585618 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.585469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38073 (local address 127.0.0.1:48538)
0619 16:51:54.585519 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:54.585553 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 16:51:54.585793 13781 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38073: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38073: connect: Connection refused (error 111)
I20260619 16:51:54.593712 13863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:51:54.596328 13863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:51:54.596470 13863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:51:54.601146 13863 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38073
I20260619 16:51:54.601181 13930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38073 every 8 connection(s)
I20260619 16:51:54.601580 13863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-2/data/info.pb
I20260619 16:51:54.602631 13931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:51:54.604507 13931 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36589 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:51:54.608577 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 13863
I20260619 16:51:54.608703 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781887741903642-392-0/minicluster-data/master-2/wal/instance
I20260619 16:51:54.608950 392 external_mini_cluster.cc:1428] Setting key 14ff3876fe717505fca2084f660b00ca
Jun 19 16:51:54 dist-test-slave-jsw9 krb5kdc[13700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:51:54.614324 13784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.605397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37840 (local address 127.0.98.62:36589)
0619 16:51:54.605462 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:54.605464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:54.605644 (+ 180us) server_negotiation.cc:407] Connection header received
0619 16:51:54.606157 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:54.606158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:54.606168 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:54.606261 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:54.606972 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.607533 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.608614 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.608902 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.608981 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:54.611357 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:54.611359 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:54.611359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:54.611369 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:54.612887 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:54.613310 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:54.613311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:54.613311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:54.613379 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:54.613605 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:54.613606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:54.613606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:54.613731 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:54.613789 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:54.614114 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 16:51:54.614241 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
Jun 19 16:51:54 dist-test-slave-jsw9 krb5kdc[13700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 16:51:54.614564 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.605324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36589 (local address 127.0.0.1:37840)
0619 16:51:54.605510 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:54.605560 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:54.606072 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:54.606334 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:54.606337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:54.606631 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:51:54.606870 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:54.606875 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.607634 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.607636 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:54.608524 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:54.608527 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.608983 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.608984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:54.609132 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:54.609713 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:51:54.609723 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:51:54.611265 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:51:54.612942 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:54.612949 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:54.612952 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:54.613235 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:54.613436 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:54.613438 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:54.613438 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:54.613549 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:54.613786 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:51:54.613792 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:51:54.613953 (+ 161us) client_negotiation.cc:769] Sending connection context
0619 16:51:54.614077 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 16:51:54.614282 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":82,"threads_started":1}
I20260619 16:51:54.615145 13931 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36603 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:51:54.617723 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.609370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48540 (local address 127.0.98.60:38073)
0619 16:51:54.609735 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:54.609739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:54.609757 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:51:54.609775 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:54.609777 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:54.609795 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:54.609868 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:54.610551 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.611350 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.612146 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.612589 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.612678 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:54.615094 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:54.615102 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:54.615102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:54.615110 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:54.616593 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:54.616925 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:54.616928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:54.616929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:54.616987 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:54.617151 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:54.617152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:54.617153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:54.617301 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:54.617442 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:54.617517 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 16:51:54.617601 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":81,"threads_started":1}
Jun 19 16:51:54 dist-test-slave-jsw9 krb5kdc[13700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 16:51:54.621773 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.615433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36603 (local address 127.0.0.1:33406)
0619 16:51:54.615572 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:54.615583 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:54.615632 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:54.615760 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:54.615761 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:54.615898 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:51:54.615953 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:54.615954 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.616584 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.616584 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:54.617076 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:54.617077 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.617417 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.617418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:54.617486 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:54.617879 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:51:54.617884 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:51:54.619185 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:51:54.620624 (+ 1439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:54.620625 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:54.620625 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:54.620780 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:54.621157 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:54.621158 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:54.621158 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:54.621213 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:54.621454 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:51:54.621455 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:51:54.621502 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 16:51:54.621561 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 16:51:54.621666 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260619 16:51:54.621773 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.615512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33406 (local address 127.0.98.61:36603)
0619 16:51:54.615566 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:54.615568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:54.615616 (+ 48us) server_negotiation.cc:407] Connection header received
0619 16:51:54.615691 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:54.615692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:54.615703 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:54.615756 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:54.616023 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.616516 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.617148 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.617360 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.617418 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:54.619271 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:54.619273 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:54.619274 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:54.619282 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:54.620546 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:54.620844 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:54.620845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:54.620845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:54.620897 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:54.621275 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:54.621276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:54.621277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:54.621398 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:54.621456 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:54.621591 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 16:51:54.621667 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
Jun 19 16:51:54 dist-test-slave-jsw9 krb5kdc[13700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:51:54.622434 13931 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38073 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:51:54.625321 13784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.618648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37848 (local address 127.0.98.62:36589)
0619 16:51:54.618714 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:54.618715 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:54.618723 (+ 8us) server_negotiation.cc:407] Connection header received
0619 16:51:54.618746 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:54.618747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:54.618758 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:54.618803 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:54.619310 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.619802 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.620461 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.620662 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.620746 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:54.622952 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:54.622953 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:54.622954 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:54.622960 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:54.624274 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:54.624565 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:54.624566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:54.624566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:54.624624 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:54.624803 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:54.624803 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:54.624804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:54.624936 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:54.625115 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:54.625143 (+ 28us) server_negotiation.cc:299] Negotiation successful
0619 16:51:54.625218 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Jun 19 16:51:54 dist-test-slave-jsw9 krb5kdc[13700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 16:51:54.630054 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.622729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38073 (local address 127.0.0.1:48552)
0619 16:51:54.622828 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:54.622839 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:54.622891 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:54.623235 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:54.623236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:54.623371 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:51:54.623416 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:54.623418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.624193 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.624194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:54.624853 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:54.624855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.625246 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.625247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:54.625306 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:54.625784 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:51:54.625789 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:51:54.627351 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:51:54.629087 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:54.629088 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:54.629088 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:54.629259 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:54.629488 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:54.629488 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:54.629488 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:54.629540 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:54.629794 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:51:54.629794 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:51:54.629842 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 16:51:54.629889 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 16:51:54.629976 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 16:51:54.630144 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.622788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48552 (local address 127.0.98.60:38073)
0619 16:51:54.622825 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:54.622826 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:54.622875 (+ 49us) server_negotiation.cc:407] Connection header received
0619 16:51:54.623160 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:54.623161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:54.623173 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:54.623220 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:54.623492 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.624109 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.624911 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.625172 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.625238 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:54.627442 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:54.627443 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:54.627443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:54.627450 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:54.628995 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:54.629354 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:54.629355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:54.629356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:54.629419 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:54.629596 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:54.629597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:54.629597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:54.629740 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:54.629809 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:54.629933 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 16:51:54.630031 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":18}
Jun 19 16:51:54 dist-test-slave-jsw9 krb5kdc[13700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 16:51:54.632015 13931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dabd470e02944a5a667495276b1a053: Bootstrap starting.
I20260619 16:51:54.633100 13931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2dabd470e02944a5a667495276b1a053: Neither blocks nor log segments found. Creating new log.
I20260619 16:51:54.633446 13931 log.cc:824] T 00000000000000000000000000000000 P 2dabd470e02944a5a667495276b1a053: Log is configured to *not* fsync() on all Append() calls
I20260619 16:51:54.633754 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.626063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33420 (local address 127.0.98.61:36603)
0619 16:51:54.626112 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:54.626115 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:54.626123 (+ 8us) server_negotiation.cc:407] Connection header received
0619 16:51:54.626147 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:54.626148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:54.626159 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:54.626299 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:54.626821 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.627440 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.628379 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.628632 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.628694 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:54.631237 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:54.631238 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:54.631239 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:54.631246 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:54.632582 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:54.632965 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:54.632966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:54.632966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:54.633023 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:54.633253 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:54.633258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:54.633258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:54.633387 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:54.633549 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:54.633606 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 16:51:54.633680 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 16:51:54.634526 13931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dabd470e02944a5a667495276b1a053: No bootstrap required, opened a new log
I20260619 16:51:54.635725 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:51:54.636543 13931 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2dabd470e02944a5a667495276b1a053 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6737f0a071645768c81f32251f69c4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36589 } } peers { permanent_uuid: "dc8a602d13d14e718be4332560308f5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36603 } } peers { permanent_uuid: "2dabd470e02944a5a667495276b1a053" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38073 } }
I20260619 16:51:54.636689 13931 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2dabd470e02944a5a667495276b1a053 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:51:54.636723 13931 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2dabd470e02944a5a667495276b1a053 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dabd470e02944a5a667495276b1a053, State: Initialized, Role: FOLLOWER
I20260619 16:51:54.637225 13931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dabd470e02944a5a667495276b1a053 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6737f0a071645768c81f32251f69c4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36589 } } peers { permanent_uuid: "dc8a602d13d14e718be4332560308f5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36603 } } peers { permanent_uuid: "2dabd470e02944a5a667495276b1a053" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38073 } }
I20260619 16:51:54.637660 13931 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dabd470e02944a5a667495276b1a053 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:51:54.639571 13942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dabd470e02944a5a667495276b1a053 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6737f0a071645768c81f32251f69c4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36589 } } peers { permanent_uuid: "dc8a602d13d14e718be4332560308f5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36603 } } peers { permanent_uuid: "2dabd470e02944a5a667495276b1a053" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38073 } } }
I20260619 16:51:54.639704 13942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dabd470e02944a5a667495276b1a053 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:51:54.640318 13863 master_runner.cc:186] Error getting master registration for 127.0.98.62:36589: OK, instance_id { permanent_uuid: "d6737f0a071645768c81f32251f69c4a" instance_seqno: 1781887914233175 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 16:51:54.643744 13863 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 16:51:54.645327 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.637143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33428 (local address 127.0.98.61:36603)
0619 16:51:54.637188 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:54.637190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:54.637295 (+ 105us) server_negotiation.cc:407] Connection header received
0619 16:51:54.637357 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:54.637358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:54.637369 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:54.637413 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:54.638127 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.638906 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.640207 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.640519 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.640558 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:54.642129 (+ 1571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:54.642131 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:54.642131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:54.642138 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:54.643697 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:54.644064 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:54.644066 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:54.644066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:54.644128 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:54.644646 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:54.644647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:54.644647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:54.644808 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:54.644922 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:54.645131 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 16:51:54.645226 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 16:51:54.645738 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.637797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48566 (local address 127.0.98.60:38073)
0619 16:51:54.637841 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:54.637843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:54.638057 (+ 214us) server_negotiation.cc:407] Connection header received
0619 16:51:54.638144 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:54.638145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:54.638155 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:54.638181 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:54.639050 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.639765 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.640733 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.640998 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.641034 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:54.642468 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:54.642469 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:54.642470 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:54.642476 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:54.643976 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:54.644339 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:54.644340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:54.644341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:54.644404 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:54.645125 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:54.645126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:54.645126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:54.645300 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:54.645372 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:54.645563 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 16:51:54.645649 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260619 16:51:54.646147 13955 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2dabd470e02944a5a667495276b1a053: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:51:54.646273 13955 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:51:54.646910 13784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.637432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37860 (local address 127.0.98.62:36589)
0619 16:51:54.637482 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:54.637484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:54.637941 (+ 457us) server_negotiation.cc:407] Connection header received
0619 16:51:54.637996 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:54.637997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:54.638010 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:54.638058 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:54.638711 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.639403 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.640596 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.641164 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.641205 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:54.643611 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:54.643613 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:54.643613 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:54.643621 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:54.645477 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:54.646107 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:54.646108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:54.646108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:54.646171 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:54.646375 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:54.646376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:54.646376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:54.646495 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:54.646588 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:54.646726 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 16:51:54.646805 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 16:51:54.660401 13856 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38073 } attempt: 3
Jun 19 16:51:54 dist-test-slave-jsw9 krb5kdc[13700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 16:51:54.668840 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.660932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48568 (local address 127.0.98.60:38073)
0619 16:51:54.660989 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:54.660992 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:54.661002 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:51:54.661030 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:54.661031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:54.661044 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:54.661112 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:54.661495 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.662335 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.663090 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.663443 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.663515 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:54.665823 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:54.665825 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:54.665826 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:54.665834 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:54.667553 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:54.667959 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:54.667960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:54.667961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:54.668024 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:54.668230 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:54.668231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:54.668231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:54.668379 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:54.668446 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:54.668641 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 16:51:54.668728 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 16:51:54.668859 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.660839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38073 (local address 127.0.0.1:48568)
0619 16:51:54.660898 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:54.660914 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:54.660964 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:54.661178 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:54.661179 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:54.661359 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:51:54.661418 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:54.661420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.662419 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.662420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:54.663010 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:54.663013 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.663533 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.663534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:54.663600 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:54.664173 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:51:54.664178 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:51:54.665724 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:51:54.667678 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:54.667680 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:54.667680 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:54.667885 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:54.668103 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:54.668104 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:54.668104 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:54.668173 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:54.668458 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:51:54.668459 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:51:54.668523 (+ 64us) client_negotiation.cc:769] Sending connection context
0619 16:51:54.668590 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 16:51:54.668706 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 16:51:54.671264 13856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc8a602d13d14e718be4332560308f5f: Bootstrap starting.
I20260619 16:51:54.672012 13856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc8a602d13d14e718be4332560308f5f: Neither blocks nor log segments found. Creating new log.
I20260619 16:51:54.672322 13856 log.cc:824] T 00000000000000000000000000000000 P dc8a602d13d14e718be4332560308f5f: Log is configured to *not* fsync() on all Append() calls
I20260619 16:51:54.673128 13856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc8a602d13d14e718be4332560308f5f: No bootstrap required, opened a new log
I20260619 16:51:54.674846 13856 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc8a602d13d14e718be4332560308f5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6737f0a071645768c81f32251f69c4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36589 } } peers { permanent_uuid: "dc8a602d13d14e718be4332560308f5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36603 } } peers { permanent_uuid: "2dabd470e02944a5a667495276b1a053" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38073 } }
I20260619 16:51:54.674995 13856 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc8a602d13d14e718be4332560308f5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:51:54.675029 13856 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc8a602d13d14e718be4332560308f5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc8a602d13d14e718be4332560308f5f, State: Initialized, Role: FOLLOWER
I20260619 16:51:54.675168 13856 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc8a602d13d14e718be4332560308f5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6737f0a071645768c81f32251f69c4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36589 } } peers { permanent_uuid: "dc8a602d13d14e718be4332560308f5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36603 } } peers { permanent_uuid: "2dabd470e02944a5a667495276b1a053" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38073 } }
I20260619 16:51:54.675475 13958 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc8a602d13d14e718be4332560308f5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6737f0a071645768c81f32251f69c4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36589 } } peers { permanent_uuid: "dc8a602d13d14e718be4332560308f5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36603 } } peers { permanent_uuid: "2dabd470e02944a5a667495276b1a053" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38073 } } }
I20260619 16:51:54.675567 13958 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc8a602d13d14e718be4332560308f5f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:51:54.675566 13856 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc8a602d13d14e718be4332560308f5f [sys.catalog]: configured and running, proceeding with master startup.
W20260619 16:51:54.678036 13969 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dc8a602d13d14e718be4332560308f5f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:51:54.678102 13969 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:51:54.678169 13788 master_runner.cc:186] Error getting master registration for 127.0.98.62:36589: OK, instance_id { permanent_uuid: "d6737f0a071645768c81f32251f69c4a" instance_seqno: 1781887914233175 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 16:51:54.681632 13788 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 16:51:54.684988 13781 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38073 } attempt: 3
Jun 19 16:51:54 dist-test-slave-jsw9 krb5kdc[13700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 16:51:54.694278 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.685459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38073 (local address 127.0.0.1:48580)
0619 16:51:54.685537 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:54.685551 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:54.685600 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:54.685782 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:54.685783 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:54.686008 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:51:54.686082 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:54.686084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.687131 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.687133 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:54.688077 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:54.688082 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.688597 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.688598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:54.688673 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:54.689281 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:51:54.689287 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:51:54.691074 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:51:54.693035 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:54.693036 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:54.693037 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:54.693277 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:54.693484 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:54.693485 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:54.693485 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:54.693555 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:54.693865 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:51:54.693867 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:51:54.693929 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 16:51:54.694010 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 16:51:54.694134 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 16:51:54.694281 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:54.685553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48580 (local address 127.0.98.60:38073)
0619 16:51:54.685615 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:54.685617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:54.685629 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:51:54.685664 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:54.685665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:54.685678 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:54.685762 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:54.686173 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.687020 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.688205 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:54.688509 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:54.688579 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:54.691177 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:54.691179 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:54.691180 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:54.691189 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:54.692938 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:54.693353 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:54.693354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:54.693354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:54.693422 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:54.693613 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:54.693614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:54.693614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:54.693771 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:54.693872 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:54.694040 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 16:51:54.694134 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 16:51:54.696811 13781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6737f0a071645768c81f32251f69c4a: Bootstrap starting.
I20260619 16:51:54.697937 13781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6737f0a071645768c81f32251f69c4a: Neither blocks nor log segments found. Creating new log.
I20260619 16:51:54.698367 13781 log.cc:824] T 00000000000000000000000000000000 P d6737f0a071645768c81f32251f69c4a: Log is configured to *not* fsync() on all Append() calls
I20260619 16:51:54.699556 13781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6737f0a071645768c81f32251f69c4a: No bootstrap required, opened a new log
I20260619 16:51:54.701789 13781 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6737f0a071645768c81f32251f69c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6737f0a071645768c81f32251f69c4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36589 } } peers { permanent_uuid: "dc8a602d13d14e718be4332560308f5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36603 } } peers { permanent_uuid: "2dabd470e02944a5a667495276b1a053" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38073 } }
I20260619 16:51:54.701975 13781 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6737f0a071645768c81f32251f69c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:51:54.702011 13781 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6737f0a071645768c81f32251f69c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6737f0a071645768c81f32251f69c4a, State: Initialized, Role: FOLLOWER
I20260619 16:51:54.702231 13781 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6737f0a071645768c81f32251f69c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6737f0a071645768c81f32251f69c4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36589 } } peers { permanent_uuid: "dc8a602d13d14e718be4332560308f5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36603 } } peers { permanent_uuid: "2dabd470e02944a5a667495276b1a053" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38073 } }
I20260619 16:51:54.702581 13972 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6737f0a071645768c81f32251f69c4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6737f0a071645768c81f32251f69c4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36589 } } peers { permanent_uuid: "dc8a602d13d14e718be4332560308f5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36603 } } peers { permanent_uuid: "2dabd470e02944a5a667495276b1a053" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38073 } } }
I20260619 16:51:54.702656 13781 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6737f0a071645768c81f32251f69c4a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:51:54.702751 13972 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6737f0a071645768c81f32251f69c4a [sys.catalog]: This master's current role is: FOLLOWER
W20260619 16:51:54.704866 13983 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d6737f0a071645768c81f32251f69c4a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:51:54.704988 13983 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:51:54.707876 13713 master_runner.cc:263] No leader master found from master d6737f0a071645768c81f32251f69c4a
I20260619 16:51:54.707938 13713 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 16:51:54.730824 13734 tablet.cc:2369] T 00000000000000000000000000000000 P d6737f0a071645768c81f32251f69c4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:51:54.744793 13942 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2dabd470e02944a5a667495276b1a053 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:51:54.745007 13942 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2dabd470e02944a5a667495276b1a053 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6737f0a071645768c81f32251f69c4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36589 } } peers { permanent_uuid: "dc8a602d13d14e718be4332560308f5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36603 } } peers { permanent_uuid: "2dabd470e02944a5a667495276b1a053" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38073 } }
I20260619 16:51:54.745366 13942 leader_election.cc:288] T 00000000000000000000000000000000 P 2dabd470e02944a5a667495276b1a053 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6737f0a071645768c81f32251f69c4a (127.0.98.62:36589), dc8a602d13d14e718be4332560308f5f (127.0.98.61:36603)
I20260619 16:51:54.746181 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2dabd470e02944a5a667495276b1a053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc8a602d13d14e718be4332560308f5f" is_pre_election: true
I20260619 16:51:54.746467 13830 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dc8a602d13d14e718be4332560308f5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dabd470e02944a5a667495276b1a053 in term 0.
I20260619 16:51:54.746744 13882 leader_election.cc:302] T 00000000000000000000000000000000 P 2dabd470e02944a5a667495276b1a053 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dabd470e02944a5a667495276b1a053, dc8a602d13d14e718be4332560308f5f; no voters:
I20260619 16:51:54.747198 13755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2dabd470e02944a5a667495276b1a053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6737f0a071645768c81f32251f69c4a" is_pre_election: true
I20260619 16:51:54.747394 13755 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d6737f0a071645768c81f32251f69c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dabd470e02944a5a667495276b1a053 in term 0.
I20260619 16:51:54.747733 13942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2dabd470e02944a5a667495276b1a053 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:51:54.747812 13942 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2dabd470e02944a5a667495276b1a053 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:51:54.747843 13942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2dabd470e02944a5a667495276b1a053 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:51:54.749069 13942 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2dabd470e02944a5a667495276b1a053 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6737f0a071645768c81f32251f69c4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36589 } } peers { permanent_uuid: "dc8a602d13d14e718be4332560308f5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36603 } } peers { permanent_uuid: "2dabd470e02944a5a667495276b1a053" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38073 } }
I20260619 16:51:54.749254 13942 leader_election.cc:288] T 00000000000000000000000000000000 P 2dabd470e02944a5a667495276b1a053 [CANDIDATE]: Term 1 election: Requested vote from peers d6737f0a071645768c81f32251f69c4a (127.0.98.62:36589), dc8a602d13d14e718be4332560308f5f (127.0.98.61:36603)
I20260619 16:51:54.749612 13755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2dabd470e02944a5a667495276b1a053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6737f0a071645768c81f32251f69c4a"
I20260619 16:51:54.749735 13755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6737f0a071645768c81f32251f69c4a [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:51:54.751488 13830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2dabd470e02944a5a667495276b1a053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc8a602d13d14e718be4332560308f5f"
I20260619 16:51:54.751616 13830 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc8a602d13d14e718be4332560308f5f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:51:54.752738 13830 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dc8a602d13d14e718be4332560308f5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dabd470e02944a5a667495276b1a053 in term 1.
I20260619 16:51:54.752947 13882 leader_election.cc:302] T 00000000000000000000000000000000 P 2dabd470e02944a5a667495276b1a053 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dabd470e02944a5a667495276b1a053, dc8a602d13d14e718be4332560308f5f; no voters:
I20260619 16:51:54.753650 13942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2dabd470e02944a5a667495276b1a053 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:51:54.753865 13942 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2dabd470e02944a5a667495276b1a053 [term 1 LEADER]: Becoming Leader. State: Replica: 2dabd470e02944a5a667495276b1a053, State: Running, Role: LEADER
I20260619 16:51:54.753988 13942 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2dabd470e02944a5a667495276b1a053 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6737f0a071645768c81f32251f69c4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36589 } } peers { permanent_uuid: "dc8a602d13d14e718be4332560308f5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36603 } } peers { permanent_uuid: "2dabd470e02944a5a667495276b1a053" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38073 } }
I20260619 16:51:54.754555 13942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dabd470e02944a5a667495276b1a053 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dabd470e02944a5a667495276b1a053. Latest consensus state: current_term: 1 leader_uuid: "2dabd470e02944a5a667495276b1a053" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6737f0a071645768c81f32251f69c4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36589 } } peers { permanent_uuid: "dc8a602d13d14e718be4332560308f5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36603 } } peers { permanent_uuid: "2dabd470e02944a5a667495276b1a053" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38073 } } }
I20260619 16:51:54.754673 13942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dabd470e02944a5a667495276b1a053 [sys.catalog]: This master's current role is: LEADER
I20260619 16:51:54.755330 13755 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d6737f0a071645768c81f32251f69c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dabd470e02944a5a667495276b1a053 in term 1.
I20260619 16:51:54.756392 13987 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:51:54.756664 13987 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:51:54.759593 13830 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dc8a602d13d14e718be4332560308f5f [term 1 FOLLOWER]: Refusing update from remote peer 2dabd470e02944a5a667495276b1a053: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:51:54.759965 13755 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d6737f0a071645768c81f32251f69c4a [term 1 FOLLOWER]: Refusing update from remote peer 2dabd470e02944a5a667495276b1a053: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:51:54.760329 13942 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2dabd470e02944a5a667495276b1a053 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc8a602d13d14e718be4332560308f5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:51:54.760779 13942 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2dabd470e02944a5a667495276b1a053 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6737f0a071645768c81f32251f69c4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:51:54.762450 13972 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6737f0a071645768c81f32251f69c4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dabd470e02944a5a667495276b1a053. Latest consensus state: current_term: 1 leader_uuid: "2dabd470e02944a5a667495276b1a053" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6737f0a071645768c81f32251f69c4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36589 } } peers { permanent_uuid: "dc8a602d13d14e718be4332560308f5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36603 } } peers { permanent_uuid: "2dabd470e02944a5a667495276b1a053" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38073 } } }
I20260619 16:51:54.762564 13972 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6737f0a071645768c81f32251f69c4a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:51:54.764094 13958 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc8a602d13d14e718be4332560308f5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dabd470e02944a5a667495276b1a053. Latest consensus state: current_term: 1 leader_uuid: "2dabd470e02944a5a667495276b1a053" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6737f0a071645768c81f32251f69c4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36589 } } peers { permanent_uuid: "dc8a602d13d14e718be4332560308f5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36603 } } peers { permanent_uuid: "2dabd470e02944a5a667495276b1a053" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38073 } } }
I20260619 16:51:54.764242 13958 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc8a602d13d14e718be4332560308f5f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:51:54.766911 13942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dabd470e02944a5a667495276b1a053 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2dabd470e02944a5a667495276b1a053" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6737f0a071645768c81f32251f69c4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36589 } } peers { permanent_uuid: "dc8a602d13d14e718be4332560308f5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36603 } } peers { permanent_uuid: "2dabd470e02944a5a667495276b1a053" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38073 } } }
I20260619 16:51:54.767035 13942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dabd470e02944a5a667495276b1a053 [sys.catalog]: This master's current role is: LEADER
I20260619 16:51:54.767590 13958 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc8a602d13d14e718be4332560308f5f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2dabd470e02944a5a667495276b1a053" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6737f0a071645768c81f32251f69c4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36589 } } peers { permanent_uuid: "dc8a602d13d14e718be4332560308f5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36603 } } peers { permanent_uuid: "2dabd470e02944a5a667495276b1a053" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38073 } } }
I20260619 16:51:54.767699 13958 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc8a602d13d14e718be4332560308f5f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:51:54.767930 13988 mvcc.cc:205] Tried to move back new op lower bound from 7298612898851209216 to 7298612898833354752. Current Snapshot: MvccSnapshot[applied={T|T < 7298612898851209216}]
I20260619 16:51:54.768458 13942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dabd470e02944a5a667495276b1a053 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2dabd470e02944a5a667495276b1a053" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6737f0a071645768c81f32251f69c4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36589 } } peers { permanent_uuid: "dc8a602d13d14e718be4332560308f5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36603 } } peers { permanent_uuid: "2dabd470e02944a5a667495276b1a053" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38073 } } }
I20260619 16:51:54.768558 13942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dabd470e02944a5a667495276b1a053 [sys.catalog]: This master's current role is: LEADER
I20260619 16:51:54.769065 13972 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6737f0a071645768c81f32251f69c4a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2dabd470e02944a5a667495276b1a053" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6737f0a071645768c81f32251f69c4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36589 } } peers { permanent_uuid: "dc8a602d13d14e718be4332560308f5f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36603 } } peers { permanent_uuid: "2dabd470e02944a5a667495276b1a053" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38073 } } }
I20260619 16:51:54.769186 13972 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6737f0a071645768c81f32251f69c4a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:51:54.769529 13991 mvcc.cc:205] Tried to move back new op lower bound from 7298612898851209216 to 7298612898833354752. Current Snapshot: MvccSnapshot[applied={T|T < 7298612898851209216}]
I20260619 16:51:54.769644 13989 mvcc.cc:205] Tried to move back new op lower bound from 7298612898851209216 to 7298612898833354752. Current Snapshot: MvccSnapshot[applied={T|T < 7298612898851209216}]
I20260619 16:51:54.770679 13987 catalog_manager.cc:1347] Generated new cluster ID: 0f7767f659dc451089446dde8a8ee89e
I20260619 16:51:54.770737 13987 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:51:54.830778 13987 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:51:54.831524 13987 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:51:54.835801 13987 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2dabd470e02944a5a667495276b1a053: Generated new TSK 0
I20260619 16:51:54.836254 13987 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:51:55.678655 13969 catalog_manager.cc:1259] Loaded cluster ID: 0f7767f659dc451089446dde8a8ee89e
I20260619 16:51:55.678756 13969 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dc8a602d13d14e718be4332560308f5f: loading cluster ID for follower catalog manager: success
I20260619 16:51:55.679641 13969 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dc8a602d13d14e718be4332560308f5f: acquiring CA information for follower catalog manager: success
I20260619 16:51:55.679971 13969 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dc8a602d13d14e718be4332560308f5f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:51:55.705578 13983 catalog_manager.cc:1259] Loaded cluster ID: 0f7767f659dc451089446dde8a8ee89e
I20260619 16:51:55.705673 13983 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d6737f0a071645768c81f32251f69c4a: loading cluster ID for follower catalog manager: success
I20260619 16:51:55.706614 13983 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d6737f0a071645768c81f32251f69c4a: acquiring CA information for follower catalog manager: success
I20260619 16:51:55.706924 13983 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d6737f0a071645768c81f32251f69c4a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:51:55.897874 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 13713
I20260619 16:51:55.903162 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 13788
I20260619 16:51:55.908006 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 13863
2026-06-19T16:51:55Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2933 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:51:55 dist-test-slave-jsw9 krb5kdc[14002](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:51:55 dist-test-slave-jsw9 krb5kdc[14002](info): set up 2 sockets
Jun 19 16:51:55 dist-test-slave-jsw9 krb5kdc[14002](info): commencing operation
krb5kdc: starting...
W20260619 16:51:56.902535 392 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:51:56 dist-test-slave-jsw9 krb5kdc[14002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:51:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:51:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:51:57.041438 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44855
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44855
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:51:57.132436 14015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:51:57.132714 14015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:51:57.132753 14015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:51:57.134097 14015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:51:57.134145 14015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:51:57.134164 14015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:51:57.134204 14015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:51:57.134222 14015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:51:57.136091 14015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:51:57.136420 14015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:51:57.136672 14015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:51:57.142084 14023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:51:57.142084 14021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:51:57.142242 14020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:51:57.142622 14015 server_base.cc:1034] running on GCE node
I20260619 16:51:57.142915 14015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:51:57.143277 14015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:51:57.144452 14015 hybrid_clock.cc:648] HybridClock initialized: now 1781887917144435 us; error 36 us; skew 500 ppm
Jun 19 16:51:57 dist-test-slave-jsw9 krb5kdc[14002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:51:57.147652 14015 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:51:57.148155 14015 webserver.cc:460] Webserver started at http://127.0.98.62:45867/ using document root <none> and password file <none>
I20260619 16:51:57.148391 14015 fs_manager.cc:359] Metadata directory not provided
I20260619 16:51:57.148442 14015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:51:57.148579 14015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:51:57.149768 14015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "aff7a94fc0c3458795b7bc95e33769fb"
format_stamp: "Formatted at 2026-06-19 16:51:57 on dist-test-slave-jsw9"
server_key: "326318032316ef3eda5bf68b9d6e624b"
server_key_iv: "63dc67289521d2214c6c7ac4b7e025cb"
server_key_version: "encryptionkey@0"
I20260619 16:51:57.150127 14015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "aff7a94fc0c3458795b7bc95e33769fb"
format_stamp: "Formatted at 2026-06-19 16:51:57 on dist-test-slave-jsw9"
server_key: "326318032316ef3eda5bf68b9d6e624b"
server_key_iv: "63dc67289521d2214c6c7ac4b7e025cb"
server_key_version: "encryptionkey@0"
I20260619 16:51:57.151594 14015 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:51:57.152412 14030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:57.152573 14015 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:51:57.152655 14015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "aff7a94fc0c3458795b7bc95e33769fb"
format_stamp: "Formatted at 2026-06-19 16:51:57 on dist-test-slave-jsw9"
server_key: "326318032316ef3eda5bf68b9d6e624b"
server_key_iv: "63dc67289521d2214c6c7ac4b7e025cb"
server_key_version: "encryptionkey@0"
I20260619 16:51:57.152755 14015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:51:57.181219 14015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:51:57.183696 14015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:51:57.183871 14015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:51:57.188809 14015 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44855
I20260619 16:51:57.188880 14082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44855 every 8 connection(s)
I20260619 16:51:57.189280 14015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:51:57.190363 14083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:51:57.192979 14083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aff7a94fc0c3458795b7bc95e33769fb: Bootstrap starting.
I20260619 16:51:57.193665 14083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aff7a94fc0c3458795b7bc95e33769fb: Neither blocks nor log segments found. Creating new log.
I20260619 16:51:57.193961 14083 log.cc:824] T 00000000000000000000000000000000 P aff7a94fc0c3458795b7bc95e33769fb: Log is configured to *not* fsync() on all Append() calls
I20260619 16:51:57.194823 14083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aff7a94fc0c3458795b7bc95e33769fb: No bootstrap required, opened a new log
I20260619 16:51:57.196666 14083 raft_consensus.cc:348] T 00000000000000000000000000000000 P aff7a94fc0c3458795b7bc95e33769fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff7a94fc0c3458795b7bc95e33769fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44855 } }
I20260619 16:51:57.196835 14083 raft_consensus.cc:374] T 00000000000000000000000000000000 P aff7a94fc0c3458795b7bc95e33769fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:51:57.196858 14083 raft_consensus.cc:729] T 00000000000000000000000000000000 P aff7a94fc0c3458795b7bc95e33769fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aff7a94fc0c3458795b7bc95e33769fb, State: Initialized, Role: FOLLOWER
I20260619 16:51:57.197021 14083 consensus_queue.cc:260] T 00000000000000000000000000000000 P aff7a94fc0c3458795b7bc95e33769fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff7a94fc0c3458795b7bc95e33769fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44855 } }
I20260619 16:51:57.197088 14083 raft_consensus.cc:388] T 00000000000000000000000000000000 P aff7a94fc0c3458795b7bc95e33769fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:51:57.197110 14083 raft_consensus.cc:482] T 00000000000000000000000000000000 P aff7a94fc0c3458795b7bc95e33769fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:51:57.197142 14083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aff7a94fc0c3458795b7bc95e33769fb [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:51:57.197762 14083 raft_consensus.cc:504] T 00000000000000000000000000000000 P aff7a94fc0c3458795b7bc95e33769fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff7a94fc0c3458795b7bc95e33769fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44855 } }
I20260619 16:51:57.197861 14083 leader_election.cc:302] T 00000000000000000000000000000000 P aff7a94fc0c3458795b7bc95e33769fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aff7a94fc0c3458795b7bc95e33769fb; no voters:
I20260619 16:51:57.198063 14083 leader_election.cc:288] T 00000000000000000000000000000000 P aff7a94fc0c3458795b7bc95e33769fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:51:57.198107 14086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aff7a94fc0c3458795b7bc95e33769fb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:51:57.198316 14086 raft_consensus.cc:686] T 00000000000000000000000000000000 P aff7a94fc0c3458795b7bc95e33769fb [term 1 LEADER]: Becoming Leader. State: Replica: aff7a94fc0c3458795b7bc95e33769fb, State: Running, Role: LEADER
I20260619 16:51:57.198375 14083 sys_catalog.cc:564] T 00000000000000000000000000000000 P aff7a94fc0c3458795b7bc95e33769fb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:51:57.198371 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 14015
I20260619 16:51:57.198462 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:51:57.198462 14086 consensus_queue.cc:237] T 00000000000000000000000000000000 P aff7a94fc0c3458795b7bc95e33769fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff7a94fc0c3458795b7bc95e33769fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44855 } }
I20260619 16:51:57.198678 392 external_mini_cluster.cc:1428] Setting key 18493229093cc514f071dca1b7444861
I20260619 16:51:57.199059 14087 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff7a94fc0c3458795b7bc95e33769fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aff7a94fc0c3458795b7bc95e33769fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff7a94fc0c3458795b7bc95e33769fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44855 } } }
I20260619 16:51:57.199136 14087 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff7a94fc0c3458795b7bc95e33769fb [sys.catalog]: This master's current role is: LEADER
I20260619 16:51:57.199434 14088 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff7a94fc0c3458795b7bc95e33769fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader aff7a94fc0c3458795b7bc95e33769fb. Latest consensus state: current_term: 1 leader_uuid: "aff7a94fc0c3458795b7bc95e33769fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff7a94fc0c3458795b7bc95e33769fb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44855 } } }
I20260619 16:51:57.199565 14088 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff7a94fc0c3458795b7bc95e33769fb [sys.catalog]: This master's current role is: LEADER
I20260619 16:51:57.199664 14096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:51:57.200392 14096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:51:57.204008 14096 catalog_manager.cc:1347] Generated new cluster ID: 78c5beff5fee4af0bb945baaa5099c96
I20260619 16:51:57.204082 14096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 16:51:57 dist-test-slave-jsw9 krb5kdc[14002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:51:57.211359 14099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:57.199445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37280 (local address 127.0.98.62:44855)
0619 16:51:57.199776 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:57.199788 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:57.199895 (+ 107us) server_negotiation.cc:407] Connection header received
0619 16:51:57.200190 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:57.200200 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:57.200273 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:57.200420 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:57.201363 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:57.202411 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:57.203517 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:57.204081 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:57.204252 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:57.207399 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:57.207406 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:57.207406 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:57.207425 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:57.209548 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:57.209953 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:57.209957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:57.209957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:57.210100 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:57.210309 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:57.210310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:57.210310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:57.210666 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:57.210848 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:57.210921 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 16:51:57.211074 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":68,"threads_started":1}
I20260619 16:51:57.212404 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:51:57.235136 14096 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:51:57.235954 14096 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:51:57.244834 14096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aff7a94fc0c3458795b7bc95e33769fb: Generated new TSK 0
I20260619 16:51:57.245108 14096 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:51:57.268010 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 14015
2026-06-19T16:51:57Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1358 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:51:57 dist-test-slave-jsw9 krb5kdc[14110](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:51:57 dist-test-slave-jsw9 krb5kdc[14110](info): set up 2 sockets
Jun 19 16:51:57 dist-test-slave-jsw9 krb5kdc[14110](info): commencing operation
krb5kdc: starting...
W20260619 16:51:58.238364 392 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:51:58 dist-test-slave-jsw9 krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:51:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:51:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:51:58.369863 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37483
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37483
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:51:58.465338 14123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:51:58.465638 14123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:51:58.465682 14123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:51:58.467865 14123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:51:58.467970 14123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:51:58.468000 14123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:51:58.468019 14123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:51:58.468034 14123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:51:58.470031 14123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:51:58.470393 14123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:51:58.470657 14123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:51:58.476078 14129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:51:58.476059 14131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:51:58.476301 14128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:51:58.476402 14123 server_base.cc:1034] running on GCE node
I20260619 16:51:58.476660 14123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:51:58.477002 14123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:51:58.478222 14123 hybrid_clock.cc:648] HybridClock initialized: now 1781887918478168 us; error 34 us; skew 500 ppm
Jun 19 16:51:58 dist-test-slave-jsw9 krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:51:58.481237 14123 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:51:58.481791 14123 webserver.cc:460] Webserver started at http://127.0.98.62:46495/ using document root <none> and password file <none>
I20260619 16:51:58.482053 14123 fs_manager.cc:359] Metadata directory not provided
I20260619 16:51:58.482106 14123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:51:58.482277 14123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:51:58.483523 14123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "906da5240bf9440384008a0ac35da4d3"
format_stamp: "Formatted at 2026-06-19 16:51:58 on dist-test-slave-jsw9"
server_key: "1b5a3fa26ccee2827a750b4d96bc44cd"
server_key_iv: "e4ce5376931a2b866f0b03715ed9556f"
server_key_version: "encryptionkey@0"
I20260619 16:51:58.483955 14123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "906da5240bf9440384008a0ac35da4d3"
format_stamp: "Formatted at 2026-06-19 16:51:58 on dist-test-slave-jsw9"
server_key: "1b5a3fa26ccee2827a750b4d96bc44cd"
server_key_iv: "e4ce5376931a2b866f0b03715ed9556f"
server_key_version: "encryptionkey@0"
I20260619 16:51:58.485423 14123 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 16:51:58.486356 14138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:58.486521 14123 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:51:58.486610 14123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "906da5240bf9440384008a0ac35da4d3"
format_stamp: "Formatted at 2026-06-19 16:51:58 on dist-test-slave-jsw9"
server_key: "1b5a3fa26ccee2827a750b4d96bc44cd"
server_key_iv: "e4ce5376931a2b866f0b03715ed9556f"
server_key_version: "encryptionkey@0"
I20260619 16:51:58.486686 14123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:51:58.522554 14123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:51:58.525707 14123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:51:58.525911 14123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:51:58.532101 14123 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37483
I20260619 16:51:58.532148 14190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37483 every 8 connection(s)
I20260619 16:51:58.532927 14123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:51:58.534384 14191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:51:58.536866 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 14123
I20260619 16:51:58.537000 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:51:58.537226 392 external_mini_cluster.cc:1428] Setting key 3170158846e4c8a8505f2167bc966ee7
I20260619 16:51:58.537321 14191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3: Bootstrap starting.
I20260619 16:51:58.538125 14191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3: Neither blocks nor log segments found. Creating new log.
I20260619 16:51:58.538486 14191 log.cc:824] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3: Log is configured to *not* fsync() on all Append() calls
I20260619 16:51:58.539228 14191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3: No bootstrap required, opened a new log
I20260619 16:51:58.541483 14191 raft_consensus.cc:348] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "906da5240bf9440384008a0ac35da4d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37483 } }
I20260619 16:51:58.541687 14191 raft_consensus.cc:374] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:51:58.541723 14191 raft_consensus.cc:729] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 906da5240bf9440384008a0ac35da4d3, State: Initialized, Role: FOLLOWER
I20260619 16:51:58.541877 14191 consensus_queue.cc:260] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "906da5240bf9440384008a0ac35da4d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37483 } }
I20260619 16:51:58.541960 14191 raft_consensus.cc:388] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:51:58.541989 14191 raft_consensus.cc:482] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:51:58.542029 14191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:51:58.543164 14191 raft_consensus.cc:504] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "906da5240bf9440384008a0ac35da4d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37483 } }
I20260619 16:51:58.543403 14191 leader_election.cc:302] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 906da5240bf9440384008a0ac35da4d3; no voters:
I20260619 16:51:58.543752 14191 leader_election.cc:288] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:51:58.543823 14196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:51:58.544101 14191 sys_catalog.cc:564] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:51:58.545317 14196 raft_consensus.cc:686] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [term 1 LEADER]: Becoming Leader. State: Replica: 906da5240bf9440384008a0ac35da4d3, State: Running, Role: LEADER
I20260619 16:51:58.545450 14196 consensus_queue.cc:237] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "906da5240bf9440384008a0ac35da4d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37483 } }
Jun 19 16:51:58 dist-test-slave-jsw9 krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:51:58.545989 14196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 906da5240bf9440384008a0ac35da4d3. Latest consensus state: current_term: 1 leader_uuid: "906da5240bf9440384008a0ac35da4d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "906da5240bf9440384008a0ac35da4d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37483 } } }
I20260619 16:51:58.546089 14196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [sys.catalog]: This master's current role is: LEADER
I20260619 16:51:58.546391 14205 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:51:58.546838 14197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "906da5240bf9440384008a0ac35da4d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "906da5240bf9440384008a0ac35da4d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37483 } } }
I20260619 16:51:58.546927 14197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [sys.catalog]: This master's current role is: LEADER
I20260619 16:51:58.547784 14205 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:51:58.550122 14205 catalog_manager.cc:1347] Generated new cluster ID: fd60ec7e79c6475791220ba6d70e7aae
I20260619 16:51:58.550216 14205 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:51:58.551935 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:58.537966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43264 (local address 127.0.98.62:37483)
0619 16:51:58.538170 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:58.538180 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:58.538261 (+ 81us) server_negotiation.cc:407] Connection header received
0619 16:51:58.538865 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:58.538874 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:58.539013 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:58.539161 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:58.540128 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:58.541193 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:58.542422 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:58.542948 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:58.543047 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:58.546774 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:58.546780 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:58.546780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:58.546797 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:58.549302 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:58.549835 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:58.549839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:58.549839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:58.549975 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:58.550307 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:58.550307 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:58.550308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:58.550667 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:58.551419 (+ 752us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:58.551504 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 16:51:58.551679 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":85,"threads_started":1}
I20260619 16:51:58.575078 14205 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:51:58.576030 14205 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:51:58.579723 14205 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3: Generated new TSK 0
I20260619 16:51:58.579959 14205 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:51:58.614004 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37483
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:51:58.707599 14217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:51:58.707976 14217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:51:58.708050 14217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:51:58.709532 14217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:51:58.709585 14217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:51:58.709618 14217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:51:58.711357 14217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:51:58.711606 14217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:51:58.711840 14217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:51:58.717376 14225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:51:58.717435 14217 server_base.cc:1034] running on GCE node
W20260619 16:51:58.717398 14223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:51:58.717406 14222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:51:58.717814 14217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:51:58.718152 14217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:51:58.719336 14217 hybrid_clock.cc:648] HybridClock initialized: now 1781887918719310 us; error 42 us; skew 500 ppm
Jun 19 16:51:58 dist-test-slave-jsw9 krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:51:58.722460 14217 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:51:58.722957 14217 webserver.cc:460] Webserver started at http://127.0.98.1:36533/ using document root <none> and password file <none>
I20260619 16:51:58.723196 14217 fs_manager.cc:359] Metadata directory not provided
I20260619 16:51:58.723248 14217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:51:58.723380 14217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:51:58.724653 14217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/data/instance:
uuid: "0f39a21a9554467bab74fe802df7a1b3"
format_stamp: "Formatted at 2026-06-19 16:51:58 on dist-test-slave-jsw9"
server_key: "5ee1a4fbc34fe1be61f11e088a0c78f6"
server_key_iv: "9897c7ae7c8ad362dbc3bbe0f67acb08"
server_key_version: "encryptionkey@0"
I20260619 16:51:58.725031 14217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0f39a21a9554467bab74fe802df7a1b3"
format_stamp: "Formatted at 2026-06-19 16:51:58 on dist-test-slave-jsw9"
server_key: "5ee1a4fbc34fe1be61f11e088a0c78f6"
server_key_iv: "9897c7ae7c8ad362dbc3bbe0f67acb08"
server_key_version: "encryptionkey@0"
I20260619 16:51:58.726460 14217 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 16:51:58.727339 14232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:58.727562 14217 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:51:58.727661 14217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/wal
uuid: "0f39a21a9554467bab74fe802df7a1b3"
format_stamp: "Formatted at 2026-06-19 16:51:58 on dist-test-slave-jsw9"
server_key: "5ee1a4fbc34fe1be61f11e088a0c78f6"
server_key_iv: "9897c7ae7c8ad362dbc3bbe0f67acb08"
server_key_version: "encryptionkey@0"
I20260619 16:51:58.727728 14217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:51:58.751122 14217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:51:58.753533 14217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:51:58.753698 14217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:51:58.753966 14217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:51:58.754344 14217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:51:58.754390 14217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:58.754423 14217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:51:58.754447 14217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:58.761412 14217 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34561
I20260619 16:51:58.761466 14345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34561 every 8 connection(s)
I20260619 16:51:58.761845 14217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 16:51:58 dist-test-slave-jsw9 krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:51:58.770498 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 14217
I20260619 16:51:58.770624 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:51:58.770854 392 external_mini_cluster.cc:1428] Setting key 74cb8ed1e965cb944bdb3422a02652dc
I20260619 16:51:58.772734 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:58.762785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37625 (local address 127.0.98.62:37483)
0619 16:51:58.762845 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:58.762848 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:58.763152 (+ 304us) server_negotiation.cc:407] Connection header received
0619 16:51:58.763719 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:58.763720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:58.763731 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:58.763804 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:58.764946 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:58.765530 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:58.766757 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:58.766985 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:58.767049 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:58.769685 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:58.769686 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:58.769687 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:58.769695 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:58.771261 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:58.771733 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:58.771735 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:58.771735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:58.771799 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:58.772032 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:58.772032 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:58.772033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:58.772158 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:58.772214 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:58.772522 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 16:51:58.772635 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 16:51:58.773022 14348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:58.762752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.1:37625)
0619 16:51:58.762931 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:58.762980 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:58.763560 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:58.763885 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:58.763889 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:58.764565 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:51:58.764808 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:58.764814 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:58.765651 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:58.765652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:58.766637 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:58.766641 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:58.767093 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:58.767094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:58.767262 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:58.767921 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:51:58.767932 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:51:58.769553 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:51:58.771347 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:58.771357 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:58.771360 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:58.771669 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:58.771857 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:58.771858 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:58.771858 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:58.771976 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:58.772213 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:51:58.772222 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:51:58.772394 (+ 172us) client_negotiation.cc:769] Sending connection context
0619 16:51:58.772492 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 16:51:58.772684 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":76,"threads_started":1}
I20260619 16:51:58.773772 14346 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37483
I20260619 16:51:58.773885 14346 heartbeater.cc:464] Registering TS with master...
I20260619 16:51:58.774103 14346 heartbeater.cc:511] Master 127.0.98.62:37483 requested a full tablet report, sending...
I20260619 16:51:58.775018 14155 ts_manager.cc:194] Registered new tserver with Master: 0f39a21a9554467bab74fe802df7a1b3 (127.0.98.1:34561)
I20260619 16:51:58.775938 14155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37625
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:51:58.797046 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37483
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:51:58.889004 14351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:51:58.889294 14351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:51:58.889331 14351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:51:58.890677 14351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:51:58.890733 14351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:51:58.890771 14351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:51:58.892323 14351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:51:58.892592 14351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:51:58.892848 14351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:51:58.898228 14357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:51:58.898244 14359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:51:58.898242 14356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:51:58.899382 14351 server_base.cc:1034] running on GCE node
I20260619 16:51:58.899712 14351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:51:58.900033 14351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:51:58.901203 14351 hybrid_clock.cc:648] HybridClock initialized: now 1781887918901185 us; error 39 us; skew 500 ppm
Jun 19 16:51:58 dist-test-slave-jsw9 krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:51:58.904400 14351 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:51:58.905129 14351 webserver.cc:460] Webserver started at http://127.0.98.2:37585/ using document root <none> and password file <none>
I20260619 16:51:58.905373 14351 fs_manager.cc:359] Metadata directory not provided
I20260619 16:51:58.905429 14351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:51:58.905656 14351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:51:58.906980 14351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/data/instance:
uuid: "6a666372e1ed43b9925eb0d2a61e4891"
format_stamp: "Formatted at 2026-06-19 16:51:58 on dist-test-slave-jsw9"
server_key: "83382d978b9301a71f392c15cd6a590c"
server_key_iv: "d3ef65d94487ffc4a731d7550acf2e66"
server_key_version: "encryptionkey@0"
I20260619 16:51:58.907364 14351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6a666372e1ed43b9925eb0d2a61e4891"
format_stamp: "Formatted at 2026-06-19 16:51:58 on dist-test-slave-jsw9"
server_key: "83382d978b9301a71f392c15cd6a590c"
server_key_iv: "d3ef65d94487ffc4a731d7550acf2e66"
server_key_version: "encryptionkey@0"
I20260619 16:51:58.908938 14351 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 16:51:58.909864 14366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:58.910063 14351 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:51:58.910154 14351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/wal
uuid: "6a666372e1ed43b9925eb0d2a61e4891"
format_stamp: "Formatted at 2026-06-19 16:51:58 on dist-test-slave-jsw9"
server_key: "83382d978b9301a71f392c15cd6a590c"
server_key_iv: "d3ef65d94487ffc4a731d7550acf2e66"
server_key_version: "encryptionkey@0"
I20260619 16:51:58.910247 14351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:51:58.929294 14351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:51:58.931821 14351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:51:58.931967 14351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:51:58.932215 14351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:51:58.932540 14351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:51:58.932570 14351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:58.932595 14351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:51:58.932607 14351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:58.939266 14351 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37581
I20260619 16:51:58.939337 14479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37581 every 8 connection(s)
I20260619 16:51:58.939705 14351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:51:58.942911 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 14351
I20260619 16:51:58.943035 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:51:58.943262 392 external_mini_cluster.cc:1428] Setting key a91207bda1b92b8d3513063fe7407326
Jun 19 16:51:58 dist-test-slave-jsw9 krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:51:58.950977 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:58.940651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53129 (local address 127.0.98.62:37483)
0619 16:51:58.940751 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:58.940753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:58.940954 (+ 201us) server_negotiation.cc:407] Connection header received
0619 16:51:58.941698 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:58.941700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:58.941715 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:58.941773 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:58.942955 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:58.943643 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:58.944810 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:58.945064 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:58.945128 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:58.947764 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:58.947766 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:58.947766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:58.947775 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:58.949439 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:58.949899 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:58.949900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:58.949900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:58.949966 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:58.950249 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:58.950250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:58.950251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:58.950388 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:58.950449 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:58.950766 (+ 317us) server_negotiation.cc:299] Negotiation successful
0619 16:51:58.950866 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 16:51:58.951184 14482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:58.940563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.2:53129)
0619 16:51:58.940808 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:58.940863 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:58.941549 (+ 686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:58.941863 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:58.941868 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:58.942598 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:51:58.942854 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:58.942861 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:58.943742 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:58.943742 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:58.944733 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:58.944737 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:58.945141 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:58.945142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:58.945272 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:58.945895 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:51:58.945924 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:51:58.947645 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:51:58.949536 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:58.949544 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:58.949547 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:58.949810 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:58.950025 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:58.950026 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:58.950026 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:58.950150 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:58.950446 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:51:58.950454 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:51:58.950622 (+ 168us) client_negotiation.cc:769] Sending connection context
0619 16:51:58.950727 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 16:51:58.950926 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":131,"threads_started":1}
I20260619 16:51:58.951900 14480 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37483
I20260619 16:51:58.952015 14480 heartbeater.cc:464] Registering TS with master...
I20260619 16:51:58.952239 14480 heartbeater.cc:511] Master 127.0.98.62:37483 requested a full tablet report, sending...
I20260619 16:51:58.952704 14155 ts_manager.cc:194] Registered new tserver with Master: 6a666372e1ed43b9925eb0d2a61e4891 (127.0.98.2:37581)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 16:51:58.953245 14155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53129
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:51:58.973985 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37483
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:51:59.066957 14485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:51:59.067227 14485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:51:59.067255 14485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:51:59.068545 14485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:51:59.068593 14485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:51:59.068622 14485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:51:59.070250 14485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:51:59.070526 14485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:51:59.070783 14485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:51:59.076275 14490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:51:59.076269 14493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:51:59.076413 14485 server_base.cc:1034] running on GCE node
W20260619 16:51:59.076530 14491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:51:59.076817 14485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:51:59.077117 14485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:51:59.078270 14485 hybrid_clock.cc:648] HybridClock initialized: now 1781887919078249 us; error 39 us; skew 500 ppm
Jun 19 16:51:59 dist-test-slave-jsw9 krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:51:59.081475 14485 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:51:59.082005 14485 webserver.cc:460] Webserver started at http://127.0.98.3:37129/ using document root <none> and password file <none>
I20260619 16:51:59.082306 14485 fs_manager.cc:359] Metadata directory not provided
I20260619 16:51:59.082361 14485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:51:59.082494 14485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:51:59.083666 14485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/data/instance:
uuid: "1e4b5295a038445080cc6ec2e3b5ce1d"
format_stamp: "Formatted at 2026-06-19 16:51:59 on dist-test-slave-jsw9"
server_key: "59c29fdf1cb0dcd2c7299cb792c96ec9"
server_key_iv: "b1d1214912df57469b138d88f8c65dbc"
server_key_version: "encryptionkey@0"
I20260619 16:51:59.084022 14485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1e4b5295a038445080cc6ec2e3b5ce1d"
format_stamp: "Formatted at 2026-06-19 16:51:59 on dist-test-slave-jsw9"
server_key: "59c29fdf1cb0dcd2c7299cb792c96ec9"
server_key_iv: "b1d1214912df57469b138d88f8c65dbc"
server_key_version: "encryptionkey@0"
I20260619 16:51:59.085443 14485 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 16:51:59.086339 14500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:59.086494 14485 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:51:59.086588 14485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/wal
uuid: "1e4b5295a038445080cc6ec2e3b5ce1d"
format_stamp: "Formatted at 2026-06-19 16:51:59 on dist-test-slave-jsw9"
server_key: "59c29fdf1cb0dcd2c7299cb792c96ec9"
server_key_iv: "b1d1214912df57469b138d88f8c65dbc"
server_key_version: "encryptionkey@0"
I20260619 16:51:59.086659 14485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:51:59.106181 14485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:51:59.108963 14485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:51:59.109143 14485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:51:59.109436 14485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:51:59.109825 14485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:51:59.109861 14485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:59.109895 14485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:51:59.109918 14485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:59.116621 14485 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40877
I20260619 16:51:59.116659 14613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40877 every 8 connection(s)
I20260619 16:51:59.117038 14485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:51:59.120491 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 14485
I20260619 16:51:59.120589 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:51:59.120805 392 external_mini_cluster.cc:1428] Setting key 73e8b5f5369af6f8ed03b69db8e344e3
Jun 19 16:51:59 dist-test-slave-jsw9 krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:51:59.127386 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:59.117854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40615 (local address 127.0.98.62:37483)
0619 16:51:59.117916 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:59.117919 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:59.118085 (+ 166us) server_negotiation.cc:407] Connection header received
0619 16:51:59.118725 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:59.118726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:59.118738 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:59.118803 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:59.120018 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.120631 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.121886 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.122100 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.122160 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:59.124550 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:59.124553 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:59.124553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:59.124579 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:59.125991 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:59.126459 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:59.126459 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:59.126460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:59.126511 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:59.126810 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:59.126810 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:59.126811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:59.126928 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:59.126976 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:59.127223 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 16:51:59.127308 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 16:51:59.127600 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:59.117759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.3:40615)
0619 16:51:59.117943 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:59.117992 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:59.118597 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:59.118880 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:59.118885 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:59.119614 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:51:59.119900 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:59.119908 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.120781 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.120782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:59.121816 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:59.121820 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.122165 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.122166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:59.122314 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:59.122900 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:51:59.122910 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:51:59.124444 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:51:59.126082 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:59.126090 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:59.126092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:59.126399 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:59.126591 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:59.126592 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:59.126592 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:59.126746 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:59.126973 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:51:59.126977 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:51:59.127117 (+ 140us) client_negotiation.cc:769] Sending connection context
0619 16:51:59.127200 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 16:51:59.127348 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":86,"threads_started":1}
I20260619 16:51:59.128103 14614 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37483
I20260619 16:51:59.128204 14614 heartbeater.cc:464] Registering TS with master...
I20260619 16:51:59.128381 14614 heartbeater.cc:511] Master 127.0.98.62:37483 requested a full tablet report, sending...
I20260619 16:51:59.128793 14155 ts_manager.cc:194] Registered new tserver with Master: 1e4b5295a038445080cc6ec2e3b5ce1d (127.0.98.3:40877)
I20260619 16:51:59.129168 14155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40615
I20260619 16:51:59.132509 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:51:59.139506 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:59.133973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43266 (local address 127.0.98.62:37483)
0619 16:51:59.134015 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:59.134016 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:59.134106 (+ 90us) server_negotiation.cc:407] Connection header received
0619 16:51:59.134151 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:59.134152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:59.134160 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:59.134266 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:59.134829 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.135340 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.136032 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.136224 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.136280 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:59.137410 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:59.137412 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:59.137412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:59.137419 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:59.138601 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:59.138886 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:59.138886 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:59.138887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:59.138931 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:59.139051 (+ 120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:59.139051 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:59.139052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:59.139191 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:59.139267 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:59.139365 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 16:51:59.139436 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 16:51:59.141031 14155 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43266:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:51:59.142935 14155 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:51:59.149817 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:59.145472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34561 (local address 127.0.0.1:54616)
0619 16:51:59.145587 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:59.145629 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:59.145709 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:59.146239 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:59.146240 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:59.146253 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:51:59.146332 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:59.146334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.147767 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.147768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:59.148819 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:59.148821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.149497 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.149498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:59.149576 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:59.149590 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 16:51:59.149658 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 16:51:59.149729 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":22,"thread_start_us":67,"threads_started":1}
I20260619 16:51:59.149827 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:59.145749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54616 (local address 127.0.98.1:34561)
0619 16:51:59.145973 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:59.145979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:59.145990 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:51:59.146073 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:59.146076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:59.146095 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:59.146438 (+ 343us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:51:59.146531 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.147659 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.148890 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.149411 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.149477 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:59.149547 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:59.149680 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 16:51:59.149729 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":97,"threads_started":1}
I20260619 16:51:59.150003 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:59.145474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37581 (local address 127.0.0.1:43816)
0619 16:51:59.145872 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:59.145881 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:59.145925 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:59.146181 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:59.146206 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:59.146208 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:51:59.146304 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:59.146307 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.147427 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.147428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:59.148682 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:59.148684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.149691 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.149692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:59.149780 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:59.149783 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:51:59.149823 (+ 40us) client_negotiation.cc:240] Negotiation successful
0619 16:51:59.149866 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":25,"thread_start_us":103,"threads_started":1}
I20260619 16:51:59.150045 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:59.145736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43816 (local address 127.0.98.2:37581)
0619 16:51:59.145973 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:59.145979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:59.145990 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:51:59.146036 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:59.146039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:59.146059 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:59.146137 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:51:59.146418 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.147325 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.148775 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.149524 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.149599 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:59.149656 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:59.149908 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 16:51:59.149959 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":112,"threads_started":1}
I20260619 16:51:59.150257 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:59.145688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40877 (local address 127.0.0.1:58958)
0619 16:51:59.146030 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:59.146039 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:59.146076 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:59.146579 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:59.146580 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:59.146582 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:51:59.146621 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:59.146623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.147903 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.147904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:59.148768 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:59.148771 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.149691 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.149692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:59.149774 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:59.149778 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:51:59.149823 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 16:51:59.149863 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":2,"thread_start_us":112,"threads_started":1}
I20260619 16:51:59.150548 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:59.146015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58958 (local address 127.0.98.3:40877)
0619 16:51:59.146364 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:59.146368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:59.146378 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:51:59.146419 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:59.146421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:59.146474 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:59.146572 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:51:59.146718 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.147806 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.148857 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.149552 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.149606 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:59.149656 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:59.150412 (+ 756us) server_negotiation.cc:299] Negotiation successful
0619 16:51:59.150461 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":84,"threads_started":1}
I20260619 16:51:59.150941 14414 tablet_service.cc:1467] Processing CreateTablet for tablet 9fba78bc408d4366ad098162d2a85d67 (DEFAULT_TABLE table=test-table [id=ff5a789f4b35414ebb4904358bb468b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:51:59.150940 14280 tablet_service.cc:1467] Processing CreateTablet for tablet 9fba78bc408d4366ad098162d2a85d67 (DEFAULT_TABLE table=test-table [id=ff5a789f4b35414ebb4904358bb468b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:51:59.151162 14548 tablet_service.cc:1467] Processing CreateTablet for tablet 9fba78bc408d4366ad098162d2a85d67 (DEFAULT_TABLE table=test-table [id=ff5a789f4b35414ebb4904358bb468b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:51:59.151360 14280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fba78bc408d4366ad098162d2a85d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:51:59.151360 14414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fba78bc408d4366ad098162d2a85d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:51:59.151443 14548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fba78bc408d4366ad098162d2a85d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:51:59.154376 14631 tablet_bootstrap.cc:492] T 9fba78bc408d4366ad098162d2a85d67 P 1e4b5295a038445080cc6ec2e3b5ce1d: Bootstrap starting.
I20260619 16:51:59.154821 14632 tablet_bootstrap.cc:492] T 9fba78bc408d4366ad098162d2a85d67 P 6a666372e1ed43b9925eb0d2a61e4891: Bootstrap starting.
I20260619 16:51:59.155081 14633 tablet_bootstrap.cc:492] T 9fba78bc408d4366ad098162d2a85d67 P 0f39a21a9554467bab74fe802df7a1b3: Bootstrap starting.
I20260619 16:51:59.155174 14631 tablet_bootstrap.cc:654] T 9fba78bc408d4366ad098162d2a85d67 P 1e4b5295a038445080cc6ec2e3b5ce1d: Neither blocks nor log segments found. Creating new log.
I20260619 16:51:59.155506 14632 tablet_bootstrap.cc:654] T 9fba78bc408d4366ad098162d2a85d67 P 6a666372e1ed43b9925eb0d2a61e4891: Neither blocks nor log segments found. Creating new log.
I20260619 16:51:59.155743 14633 tablet_bootstrap.cc:654] T 9fba78bc408d4366ad098162d2a85d67 P 0f39a21a9554467bab74fe802df7a1b3: Neither blocks nor log segments found. Creating new log.
I20260619 16:51:59.155790 14631 log.cc:824] T 9fba78bc408d4366ad098162d2a85d67 P 1e4b5295a038445080cc6ec2e3b5ce1d: Log is configured to *not* fsync() on all Append() calls
I20260619 16:51:59.156090 14632 log.cc:824] T 9fba78bc408d4366ad098162d2a85d67 P 6a666372e1ed43b9925eb0d2a61e4891: Log is configured to *not* fsync() on all Append() calls
I20260619 16:51:59.156262 14633 log.cc:824] T 9fba78bc408d4366ad098162d2a85d67 P 0f39a21a9554467bab74fe802df7a1b3: Log is configured to *not* fsync() on all Append() calls
I20260619 16:51:59.156672 14631 tablet_bootstrap.cc:492] T 9fba78bc408d4366ad098162d2a85d67 P 1e4b5295a038445080cc6ec2e3b5ce1d: No bootstrap required, opened a new log
I20260619 16:51:59.156738 14631 ts_tablet_manager.cc:1397] T 9fba78bc408d4366ad098162d2a85d67 P 1e4b5295a038445080cc6ec2e3b5ce1d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:51:59.156936 14633 tablet_bootstrap.cc:492] T 9fba78bc408d4366ad098162d2a85d67 P 0f39a21a9554467bab74fe802df7a1b3: No bootstrap required, opened a new log
I20260619 16:51:59.156997 14633 ts_tablet_manager.cc:1397] T 9fba78bc408d4366ad098162d2a85d67 P 0f39a21a9554467bab74fe802df7a1b3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:51:59.158347 14631 raft_consensus.cc:348] T 9fba78bc408d4366ad098162d2a85d67 P 1e4b5295a038445080cc6ec2e3b5ce1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f39a21a9554467bab74fe802df7a1b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34561 } } peers { permanent_uuid: "6a666372e1ed43b9925eb0d2a61e4891" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37581 } } peers { permanent_uuid: "1e4b5295a038445080cc6ec2e3b5ce1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40877 } }
I20260619 16:51:59.158515 14631 raft_consensus.cc:374] T 9fba78bc408d4366ad098162d2a85d67 P 1e4b5295a038445080cc6ec2e3b5ce1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:51:59.158550 14631 raft_consensus.cc:729] T 9fba78bc408d4366ad098162d2a85d67 P 1e4b5295a038445080cc6ec2e3b5ce1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e4b5295a038445080cc6ec2e3b5ce1d, State: Initialized, Role: FOLLOWER
I20260619 16:51:59.158500 14633 raft_consensus.cc:348] T 9fba78bc408d4366ad098162d2a85d67 P 0f39a21a9554467bab74fe802df7a1b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f39a21a9554467bab74fe802df7a1b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34561 } } peers { permanent_uuid: "6a666372e1ed43b9925eb0d2a61e4891" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37581 } } peers { permanent_uuid: "1e4b5295a038445080cc6ec2e3b5ce1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40877 } }
I20260619 16:51:59.158650 14633 raft_consensus.cc:374] T 9fba78bc408d4366ad098162d2a85d67 P 0f39a21a9554467bab74fe802df7a1b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:51:59.158680 14633 raft_consensus.cc:729] T 9fba78bc408d4366ad098162d2a85d67 P 0f39a21a9554467bab74fe802df7a1b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f39a21a9554467bab74fe802df7a1b3, State: Initialized, Role: FOLLOWER
I20260619 16:51:59.158676 14631 consensus_queue.cc:260] T 9fba78bc408d4366ad098162d2a85d67 P 1e4b5295a038445080cc6ec2e3b5ce1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f39a21a9554467bab74fe802df7a1b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34561 } } peers { permanent_uuid: "6a666372e1ed43b9925eb0d2a61e4891" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37581 } } peers { permanent_uuid: "1e4b5295a038445080cc6ec2e3b5ce1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40877 } }
I20260619 16:51:59.158802 14633 consensus_queue.cc:260] T 9fba78bc408d4366ad098162d2a85d67 P 0f39a21a9554467bab74fe802df7a1b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f39a21a9554467bab74fe802df7a1b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34561 } } peers { permanent_uuid: "6a666372e1ed43b9925eb0d2a61e4891" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37581 } } peers { permanent_uuid: "1e4b5295a038445080cc6ec2e3b5ce1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40877 } }
I20260619 16:51:59.159008 14631 ts_tablet_manager.cc:1428] T 9fba78bc408d4366ad098162d2a85d67 P 1e4b5295a038445080cc6ec2e3b5ce1d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:51:59.159027 14614 heartbeater.cc:503] Master 127.0.98.62:37483 was elected leader, sending a full tablet report...
I20260619 16:51:59.159075 14633 ts_tablet_manager.cc:1428] T 9fba78bc408d4366ad098162d2a85d67 P 0f39a21a9554467bab74fe802df7a1b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:51:59.159085 14346 heartbeater.cc:503] Master 127.0.98.62:37483 was elected leader, sending a full tablet report...
I20260619 16:51:59.159901 14632 tablet_bootstrap.cc:492] T 9fba78bc408d4366ad098162d2a85d67 P 6a666372e1ed43b9925eb0d2a61e4891: No bootstrap required, opened a new log
I20260619 16:51:59.159977 14632 ts_tablet_manager.cc:1397] T 9fba78bc408d4366ad098162d2a85d67 P 6a666372e1ed43b9925eb0d2a61e4891: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 16:51:59.161598 14632 raft_consensus.cc:348] T 9fba78bc408d4366ad098162d2a85d67 P 6a666372e1ed43b9925eb0d2a61e4891 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f39a21a9554467bab74fe802df7a1b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34561 } } peers { permanent_uuid: "6a666372e1ed43b9925eb0d2a61e4891" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37581 } } peers { permanent_uuid: "1e4b5295a038445080cc6ec2e3b5ce1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40877 } }
I20260619 16:51:59.161758 14632 raft_consensus.cc:374] T 9fba78bc408d4366ad098162d2a85d67 P 6a666372e1ed43b9925eb0d2a61e4891 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:51:59.161792 14632 raft_consensus.cc:729] T 9fba78bc408d4366ad098162d2a85d67 P 6a666372e1ed43b9925eb0d2a61e4891 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a666372e1ed43b9925eb0d2a61e4891, State: Initialized, Role: FOLLOWER
I20260619 16:51:59.161916 14632 consensus_queue.cc:260] T 9fba78bc408d4366ad098162d2a85d67 P 6a666372e1ed43b9925eb0d2a61e4891 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f39a21a9554467bab74fe802df7a1b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34561 } } peers { permanent_uuid: "6a666372e1ed43b9925eb0d2a61e4891" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37581 } } peers { permanent_uuid: "1e4b5295a038445080cc6ec2e3b5ce1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40877 } }
I20260619 16:51:59.162240 14632 ts_tablet_manager.cc:1428] T 9fba78bc408d4366ad098162d2a85d67 P 6a666372e1ed43b9925eb0d2a61e4891: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 16:51:59.162298 14480 heartbeater.cc:503] Master 127.0.98.62:37483 was elected leader, sending a full tablet report...
W20260619 16:51:59.190402 14481 tablet.cc:2369] T 9fba78bc408d4366ad098162d2a85d67 P 6a666372e1ed43b9925eb0d2a61e4891: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:51:59.262777 14347 tablet.cc:2369] T 9fba78bc408d4366ad098162d2a85d67 P 0f39a21a9554467bab74fe802df7a1b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:51:59.265419 14637 raft_consensus.cc:482] T 9fba78bc408d4366ad098162d2a85d67 P 1e4b5295a038445080cc6ec2e3b5ce1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:51:59.265563 14637 raft_consensus.cc:504] T 9fba78bc408d4366ad098162d2a85d67 P 1e4b5295a038445080cc6ec2e3b5ce1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f39a21a9554467bab74fe802df7a1b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34561 } } peers { permanent_uuid: "6a666372e1ed43b9925eb0d2a61e4891" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37581 } } peers { permanent_uuid: "1e4b5295a038445080cc6ec2e3b5ce1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40877 } }
I20260619 16:51:59.265956 14637 leader_election.cc:288] T 9fba78bc408d4366ad098162d2a85d67 P 1e4b5295a038445080cc6ec2e3b5ce1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f39a21a9554467bab74fe802df7a1b3 (127.0.98.1:34561), 6a666372e1ed43b9925eb0d2a61e4891 (127.0.98.2:37581)
I20260619 16:51:59.269057 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:59.266151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37581 (local address 127.0.98.3:51851)
0619 16:51:59.266253 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:59.266268 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:59.266323 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:59.266510 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:59.266511 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:59.266513 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:51:59.266611 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:59.266613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.267324 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.267325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:59.268183 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:59.268185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.268772 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.268774 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:59.268866 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:59.268869 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:51:59.268920 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 16:51:59.268962 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":47}
I20260619 16:51:59.269045 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:59.266274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44867 (local address 127.0.98.1:34561)
0619 16:51:59.266334 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:59.266336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:59.266460 (+ 124us) server_negotiation.cc:407] Connection header received
0619 16:51:59.266529 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:59.266530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:59.266545 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:59.266601 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:51:59.266779 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.267248 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.268283 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.268683 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.268754 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:59.268770 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:59.268922 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 16:51:59.268962 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 16:51:59.269061 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:59.266313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51851 (local address 127.0.98.2:37581)
0619 16:51:59.266368 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:59.266370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:59.266381 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:51:59.266418 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:59.266420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:59.266435 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:59.266496 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:51:59.266689 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.267231 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.268307 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.268695 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.268760 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:59.268772 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:59.268942 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 16:51:59.268989 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 16:51:59.269084 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:59.266151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34561 (local address 127.0.98.3:44867)
0619 16:51:59.266395 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:59.266405 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:59.266472 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:59.266609 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:59.266609 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:59.266611 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:51:59.266682 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:59.266684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.267333 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.267334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:59.268184 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:59.268186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.268768 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.268777 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:59.268847 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:59.268852 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:51:59.268900 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 16:51:59.268953 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":17,"thread_start_us":150,"threads_started":1}
I20260619 16:51:59.269663 14300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fba78bc408d4366ad098162d2a85d67" candidate_uuid: "1e4b5295a038445080cc6ec2e3b5ce1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f39a21a9554467bab74fe802df7a1b3" is_pre_election: true
I20260619 16:51:59.269897 14300 raft_consensus.cc:2454] T 9fba78bc408d4366ad098162d2a85d67 P 0f39a21a9554467bab74fe802df7a1b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e4b5295a038445080cc6ec2e3b5ce1d in term 0.
I20260619 16:51:59.269861 14434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fba78bc408d4366ad098162d2a85d67" candidate_uuid: "1e4b5295a038445080cc6ec2e3b5ce1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a666372e1ed43b9925eb0d2a61e4891" is_pre_election: true
I20260619 16:51:59.270031 14434 raft_consensus.cc:2454] T 9fba78bc408d4366ad098162d2a85d67 P 6a666372e1ed43b9925eb0d2a61e4891 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e4b5295a038445080cc6ec2e3b5ce1d in term 0.
I20260619 16:51:59.270162 14504 leader_election.cc:302] T 9fba78bc408d4366ad098162d2a85d67 P 1e4b5295a038445080cc6ec2e3b5ce1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f39a21a9554467bab74fe802df7a1b3, 1e4b5295a038445080cc6ec2e3b5ce1d; no voters:
I20260619 16:51:59.270373 14637 raft_consensus.cc:2781] T 9fba78bc408d4366ad098162d2a85d67 P 1e4b5295a038445080cc6ec2e3b5ce1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:51:59.270442 14637 raft_consensus.cc:482] T 9fba78bc408d4366ad098162d2a85d67 P 1e4b5295a038445080cc6ec2e3b5ce1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:51:59.270458 14637 raft_consensus.cc:3037] T 9fba78bc408d4366ad098162d2a85d67 P 1e4b5295a038445080cc6ec2e3b5ce1d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:51:59.271318 14637 raft_consensus.cc:504] T 9fba78bc408d4366ad098162d2a85d67 P 1e4b5295a038445080cc6ec2e3b5ce1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f39a21a9554467bab74fe802df7a1b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34561 } } peers { permanent_uuid: "6a666372e1ed43b9925eb0d2a61e4891" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37581 } } peers { permanent_uuid: "1e4b5295a038445080cc6ec2e3b5ce1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40877 } }
I20260619 16:51:59.271467 14637 leader_election.cc:288] T 9fba78bc408d4366ad098162d2a85d67 P 1e4b5295a038445080cc6ec2e3b5ce1d [CANDIDATE]: Term 1 election: Requested vote from peers 0f39a21a9554467bab74fe802df7a1b3 (127.0.98.1:34561), 6a666372e1ed43b9925eb0d2a61e4891 (127.0.98.2:37581)
I20260619 16:51:59.271659 14300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fba78bc408d4366ad098162d2a85d67" candidate_uuid: "1e4b5295a038445080cc6ec2e3b5ce1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f39a21a9554467bab74fe802df7a1b3"
I20260619 16:51:59.271662 14434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fba78bc408d4366ad098162d2a85d67" candidate_uuid: "1e4b5295a038445080cc6ec2e3b5ce1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a666372e1ed43b9925eb0d2a61e4891"
I20260619 16:51:59.271755 14434 raft_consensus.cc:3037] T 9fba78bc408d4366ad098162d2a85d67 P 6a666372e1ed43b9925eb0d2a61e4891 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:51:59.271806 14300 raft_consensus.cc:3037] T 9fba78bc408d4366ad098162d2a85d67 P 0f39a21a9554467bab74fe802df7a1b3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:51:59.273123 14300 raft_consensus.cc:2454] T 9fba78bc408d4366ad098162d2a85d67 P 0f39a21a9554467bab74fe802df7a1b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e4b5295a038445080cc6ec2e3b5ce1d in term 1.
I20260619 16:51:59.273123 14434 raft_consensus.cc:2454] T 9fba78bc408d4366ad098162d2a85d67 P 6a666372e1ed43b9925eb0d2a61e4891 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e4b5295a038445080cc6ec2e3b5ce1d in term 1.
I20260619 16:51:59.273365 14504 leader_election.cc:302] T 9fba78bc408d4366ad098162d2a85d67 P 1e4b5295a038445080cc6ec2e3b5ce1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f39a21a9554467bab74fe802df7a1b3, 1e4b5295a038445080cc6ec2e3b5ce1d; no voters:
I20260619 16:51:59.273484 14637 raft_consensus.cc:2781] T 9fba78bc408d4366ad098162d2a85d67 P 1e4b5295a038445080cc6ec2e3b5ce1d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:51:59.273743 14637 raft_consensus.cc:686] T 9fba78bc408d4366ad098162d2a85d67 P 1e4b5295a038445080cc6ec2e3b5ce1d [term 1 LEADER]: Becoming Leader. State: Replica: 1e4b5295a038445080cc6ec2e3b5ce1d, State: Running, Role: LEADER
I20260619 16:51:59.273867 14637 consensus_queue.cc:237] T 9fba78bc408d4366ad098162d2a85d67 P 1e4b5295a038445080cc6ec2e3b5ce1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f39a21a9554467bab74fe802df7a1b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34561 } } peers { permanent_uuid: "6a666372e1ed43b9925eb0d2a61e4891" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37581 } } peers { permanent_uuid: "1e4b5295a038445080cc6ec2e3b5ce1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40877 } }
I20260619 16:51:59.274941 14154 catalog_manager.cc:5495] T 9fba78bc408d4366ad098162d2a85d67 P 1e4b5295a038445080cc6ec2e3b5ce1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e4b5295a038445080cc6ec2e3b5ce1d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "1e4b5295a038445080cc6ec2e3b5ce1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f39a21a9554467bab74fe802df7a1b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a666372e1ed43b9925eb0d2a61e4891" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e4b5295a038445080cc6ec2e3b5ce1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40877 } health_report { overall_health: HEALTHY } } }
I20260619 16:51:59.298532 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:59.295997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58966 (local address 127.0.98.3:40877)
0619 16:51:59.296067 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:59.296069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:59.296076 (+ 7us) server_negotiation.cc:407] Connection header received
0619 16:51:59.296118 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:59.296119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:59.296146 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:59.296213 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:51:59.296361 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.296888 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.297656 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.297860 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.297920 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:59.298083 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:51:59.298277 (+ 194us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:51:59.298378 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:59.298395 (+ 17us) server_negotiation.cc:299] Negotiation successful
0619 16:51:59.298421 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 16:51:59.300828 14300 raft_consensus.cc:1261] T 9fba78bc408d4366ad098162d2a85d67 P 0f39a21a9554467bab74fe802df7a1b3 [term 1 FOLLOWER]: Refusing update from remote peer 1e4b5295a038445080cc6ec2e3b5ce1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:51:59.300828 14434 raft_consensus.cc:1261] T 9fba78bc408d4366ad098162d2a85d67 P 6a666372e1ed43b9925eb0d2a61e4891 [term 1 FOLLOWER]: Refusing update from remote peer 1e4b5295a038445080cc6ec2e3b5ce1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:51:59.301208 14641 consensus_queue.cc:1035] T 9fba78bc408d4366ad098162d2a85d67 P 1e4b5295a038445080cc6ec2e3b5ce1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f39a21a9554467bab74fe802df7a1b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:51:59.301360 14637 consensus_queue.cc:1035] T 9fba78bc408d4366ad098162d2a85d67 P 1e4b5295a038445080cc6ec2e3b5ce1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a666372e1ed43b9925eb0d2a61e4891" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:51:59.304349 14645 mvcc.cc:205] Tried to move back new op lower bound from 7298612917453135872 to 7298612917346807808. Current Snapshot: MvccSnapshot[applied={T|T < 7298612917453135872}]
I20260619 16:51:59.306794 14154 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:51:59.306975 14154 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:51:59.307828 14154 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3: Sending DeleteTablet for 3 replicas of tablet 9fba78bc408d4366ad098162d2a85d67
I20260619 16:51:59.308202 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 14217
I20260619 16:51:59.308277 14414 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fba78bc408d4366ad098162d2a85d67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:51:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43816
I20260619 16:51:59.308270 14280 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fba78bc408d4366ad098162d2a85d67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:51:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54616
I20260619 16:51:59.308887 14654 tablet_replica.cc:330] T 9fba78bc408d4366ad098162d2a85d67 P 6a666372e1ed43b9925eb0d2a61e4891: stopping tablet replica
I20260619 16:51:59.308965 14548 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fba78bc408d4366ad098162d2a85d67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:51:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58958
I20260619 16:51:59.309008 14654 raft_consensus.cc:2229] T 9fba78bc408d4366ad098162d2a85d67 P 6a666372e1ed43b9925eb0d2a61e4891 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:51:59.309124 14654 raft_consensus.cc:2258] T 9fba78bc408d4366ad098162d2a85d67 P 6a666372e1ed43b9925eb0d2a61e4891 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:51:59.309203 14655 tablet_replica.cc:330] T 9fba78bc408d4366ad098162d2a85d67 P 1e4b5295a038445080cc6ec2e3b5ce1d: stopping tablet replica
I20260619 16:51:59.309285 14655 raft_consensus.cc:2229] T 9fba78bc408d4366ad098162d2a85d67 P 1e4b5295a038445080cc6ec2e3b5ce1d [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:51:59.309424 14655 raft_consensus.cc:2258] T 9fba78bc408d4366ad098162d2a85d67 P 1e4b5295a038445080cc6ec2e3b5ce1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:51:59.309692 14654 ts_tablet_manager.cc:1905] T 9fba78bc408d4366ad098162d2a85d67 P 6a666372e1ed43b9925eb0d2a61e4891: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:51:59.309942 14655 ts_tablet_manager.cc:1905] T 9fba78bc408d4366ad098162d2a85d67 P 1e4b5295a038445080cc6ec2e3b5ce1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:51:59.311714 14655 ts_tablet_manager.cc:1918] T 9fba78bc408d4366ad098162d2a85d67 P 1e4b5295a038445080cc6ec2e3b5ce1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:51:59.311789 14655 log.cc:1196] T 9fba78bc408d4366ad098162d2a85d67 P 1e4b5295a038445080cc6ec2e3b5ce1d: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/wal/wals/9fba78bc408d4366ad098162d2a85d67
I20260619 16:51:59.311772 14654 ts_tablet_manager.cc:1918] T 9fba78bc408d4366ad098162d2a85d67 P 6a666372e1ed43b9925eb0d2a61e4891: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:51:59.311844 14654 log.cc:1196] T 9fba78bc408d4366ad098162d2a85d67 P 6a666372e1ed43b9925eb0d2a61e4891: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/wal/wals/9fba78bc408d4366ad098162d2a85d67
I20260619 16:51:59.312109 14655 ts_tablet_manager.cc:1939] T 9fba78bc408d4366ad098162d2a85d67 P 1e4b5295a038445080cc6ec2e3b5ce1d: Deleting consensus metadata
I20260619 16:51:59.312122 14654 ts_tablet_manager.cc:1939] T 9fba78bc408d4366ad098162d2a85d67 P 6a666372e1ed43b9925eb0d2a61e4891: Deleting consensus metadata
I20260619 16:51:59.312813 14141 catalog_manager.cc:4841] TS 1e4b5295a038445080cc6ec2e3b5ce1d (127.0.98.3:40877): tablet 9fba78bc408d4366ad098162d2a85d67 (table test-table [id=ff5a789f4b35414ebb4904358bb468b9]) successfully deleted
I20260619 16:51:59.312943 14141 catalog_manager.cc:4841] TS 6a666372e1ed43b9925eb0d2a61e4891 (127.0.98.2:37581): tablet 9fba78bc408d4366ad098162d2a85d67 (table test-table [id=ff5a789f4b35414ebb4904358bb468b9]) successfully deleted
W20260619 16:51:59.315820 14142 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34561 (error 108)
I20260619 16:51:59.316449 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:59.316293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34561 (local address 127.0.0.1:54630)
0619 16:51:59.316359 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:59.316406 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 16:51:59.316473 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 14351
W20260619 16:51:59.318450 14142 catalog_manager.cc:4584] TS 0f39a21a9554467bab74fe802df7a1b3 (127.0.98.1:34561): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9fba78bc408d4366ad098162d2a85d67: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34561: connect: Connection refused (error 111)
I20260619 16:51:59.321995 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 14485
I20260619 16:51:59.327502 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 14123
I20260619 16:51:59.333560 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37483
--webserver_interface=127.0.98.62
--webserver_port=46495
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:51:59.428565 14657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:51:59.428833 14657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:51:59.428855 14657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:51:59.430088 14657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:51:59.430133 14657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:51:59.430146 14657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:51:59.430159 14657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:51:59.430171 14657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:51:59.431788 14657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:51:59.432020 14657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:51:59.432237 14657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:51:59.442143 14657 server_base.cc:1034] running on GCE node
W20260619 16:51:59.442106 14666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:51:59.442080 14664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:51:59.442081 14663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:51:59.442775 14657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:51:59.443140 14657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:51:59.444299 14657 hybrid_clock.cc:648] HybridClock initialized: now 1781887919444286 us; error 31 us; skew 500 ppm
Jun 19 16:51:59 dist-test-slave-jsw9 krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:51:59.447322 14657 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:51:59.447873 14657 webserver.cc:460] Webserver started at http://127.0.98.62:46495/ using document root <none> and password file <none>
I20260619 16:51:59.448114 14657 fs_manager.cc:359] Metadata directory not provided
I20260619 16:51:59.448218 14657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:51:59.449945 14657 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:51:59.450809 14673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:59.451013 14657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 16:51:59.451118 14657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "906da5240bf9440384008a0ac35da4d3"
format_stamp: "Formatted at 2026-06-19 16:51:58 on dist-test-slave-jsw9"
server_key: "1b5a3fa26ccee2827a750b4d96bc44cd"
server_key_iv: "e4ce5376931a2b866f0b03715ed9556f"
server_key_version: "encryptionkey@0"
I20260619 16:51:59.451619 14657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:51:59.467070 14657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:51:59.469583 14657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:51:59.469758 14657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:51:59.474586 14657 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37483
I20260619 16:51:59.474617 14725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37483 every 8 connection(s)
I20260619 16:51:59.475090 14657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:51:59.477038 14726 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:51:59.477818 14726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3: Bootstrap starting.
I20260619 16:51:59.480341 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 14657
I20260619 16:51:59.480760 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34561
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37483
--webserver_port=36533
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:51:59.489178 14726 log.cc:824] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3: Log is configured to *not* fsync() on all Append() calls
I20260619 16:51:59.491631 14726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:51:59.491933 14726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3: Bootstrap complete.
I20260619 16:51:59.494443 14726 raft_consensus.cc:348] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "906da5240bf9440384008a0ac35da4d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37483 } }
I20260619 16:51:59.494722 14726 raft_consensus.cc:729] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 906da5240bf9440384008a0ac35da4d3, State: Initialized, Role: FOLLOWER
I20260619 16:51:59.494905 14726 consensus_queue.cc:260] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "906da5240bf9440384008a0ac35da4d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37483 } }
I20260619 16:51:59.494985 14726 raft_consensus.cc:388] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:51:59.495044 14726 raft_consensus.cc:482] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:51:59.495095 14726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:51:59.496351 14726 raft_consensus.cc:504] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "906da5240bf9440384008a0ac35da4d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37483 } }
I20260619 16:51:59.496465 14726 leader_election.cc:302] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 906da5240bf9440384008a0ac35da4d3; no voters:
I20260619 16:51:59.496661 14726 leader_election.cc:288] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 16:51:59.496723 14730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 16:51:59.496887 14730 raft_consensus.cc:686] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [term 2 LEADER]: Becoming Leader. State: Replica: 906da5240bf9440384008a0ac35da4d3, State: Running, Role: LEADER
I20260619 16:51:59.496979 14726 sys_catalog.cc:564] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:51:59.497044 14730 consensus_queue.cc:237] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "906da5240bf9440384008a0ac35da4d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37483 } }
I20260619 16:51:59.497455 14732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 906da5240bf9440384008a0ac35da4d3. Latest consensus state: current_term: 2 leader_uuid: "906da5240bf9440384008a0ac35da4d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "906da5240bf9440384008a0ac35da4d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37483 } } }
I20260619 16:51:59.497462 14731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "906da5240bf9440384008a0ac35da4d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "906da5240bf9440384008a0ac35da4d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37483 } } }
I20260619 16:51:59.497557 14732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [sys.catalog]: This master's current role is: LEADER
I20260619 16:51:59.497562 14731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [sys.catalog]: This master's current role is: LEADER
I20260619 16:51:59.497805 14738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:51:59.498847 14738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:51:59.498987 14738 catalog_manager.cc:1259] Loaded cluster ID: fd60ec7e79c6475791220ba6d70e7aae
I20260619 16:51:59.499058 14738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:51:59.499607 14738 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 16:51:59.504245 14738 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 16:51:59.507243 14738 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:51:59.507513 14738 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3: Loaded TSK: 0
I20260619 16:51:59.507761 14738 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 16:51:59.577481 14728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:51:59.577760 14728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:51:59.577800 14728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:51:59.579727 14728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:51:59.579810 14728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:51:59.579860 14728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:51:59.582083 14728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34561
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36533
--tserver_master_addrs=127.0.98.62:37483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:51:59.582486 14728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:51:59.582779 14728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:51:59.588739 14756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:51:59.588881 14754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:51:59.589021 14728 server_base.cc:1034] running on GCE node
W20260619 16:51:59.588742 14753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:51:59.589387 14728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:51:59.589780 14728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:51:59.590965 14728 hybrid_clock.cc:648] HybridClock initialized: now 1781887919590940 us; error 43 us; skew 500 ppm
Jun 19 16:51:59 dist-test-slave-jsw9 krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:51:59.594100 14728 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:51:59.594691 14728 webserver.cc:460] Webserver started at http://127.0.98.1:36533/ using document root <none> and password file <none>
I20260619 16:51:59.594923 14728 fs_manager.cc:359] Metadata directory not provided
I20260619 16:51:59.594990 14728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:51:59.597786 14728 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:51:59.598724 14763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:59.598901 14728 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 16:51:59.599016 14728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/wal
uuid: "0f39a21a9554467bab74fe802df7a1b3"
format_stamp: "Formatted at 2026-06-19 16:51:58 on dist-test-slave-jsw9"
server_key: "5ee1a4fbc34fe1be61f11e088a0c78f6"
server_key_iv: "9897c7ae7c8ad362dbc3bbe0f67acb08"
server_key_version: "encryptionkey@0"
I20260619 16:51:59.599496 14728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:51:59.609990 14728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:51:59.612262 14728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:51:59.612430 14728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:51:59.612680 14728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:51:59.613179 14770 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 16:51:59.614805 14728 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 16:51:59.614858 14728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 16:51:59.614886 14728 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 16:51:59.615610 14728 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 16:51:59.615653 14728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 16:51:59.615726 14770 tablet_bootstrap.cc:492] T 9fba78bc408d4366ad098162d2a85d67 P 0f39a21a9554467bab74fe802df7a1b3: Bootstrap starting.
I20260619 16:51:59.624712 14728 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34561
I20260619 16:51:59.624776 14877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34561 every 8 connection(s)
I20260619 16:51:59.625427 14728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:51:59.627702 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 14728
I20260619 16:51:59.628039 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37581
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37483
--webserver_port=37585
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:51:59.629542 14770 log.cc:824] T 9fba78bc408d4366ad098162d2a85d67 P 0f39a21a9554467bab74fe802df7a1b3: Log is configured to *not* fsync() on all Append() calls
I20260619 16:51:59.633137 14770 tablet_bootstrap.cc:492] T 9fba78bc408d4366ad098162d2a85d67 P 0f39a21a9554467bab74fe802df7a1b3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:51:59.633544 14770 tablet_bootstrap.cc:492] T 9fba78bc408d4366ad098162d2a85d67 P 0f39a21a9554467bab74fe802df7a1b3: Bootstrap complete.
I20260619 16:51:59.634871 14770 ts_tablet_manager.cc:1397] T 9fba78bc408d4366ad098162d2a85d67 P 0f39a21a9554467bab74fe802df7a1b3: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.006s
I20260619 16:51:59.636469 14770 raft_consensus.cc:348] T 9fba78bc408d4366ad098162d2a85d67 P 0f39a21a9554467bab74fe802df7a1b3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f39a21a9554467bab74fe802df7a1b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34561 } } peers { permanent_uuid: "6a666372e1ed43b9925eb0d2a61e4891" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37581 } } peers { permanent_uuid: "1e4b5295a038445080cc6ec2e3b5ce1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40877 } }
I20260619 16:51:59.636863 14770 raft_consensus.cc:729] T 9fba78bc408d4366ad098162d2a85d67 P 0f39a21a9554467bab74fe802df7a1b3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f39a21a9554467bab74fe802df7a1b3, State: Initialized, Role: FOLLOWER
I20260619 16:51:59.637053 14770 consensus_queue.cc:260] T 9fba78bc408d4366ad098162d2a85d67 P 0f39a21a9554467bab74fe802df7a1b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f39a21a9554467bab74fe802df7a1b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34561 } } peers { permanent_uuid: "6a666372e1ed43b9925eb0d2a61e4891" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37581 } } peers { permanent_uuid: "1e4b5295a038445080cc6ec2e3b5ce1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40877 } }
I20260619 16:51:59.637432 14770 ts_tablet_manager.cc:1428] T 9fba78bc408d4366ad098162d2a85d67 P 0f39a21a9554467bab74fe802df7a1b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
Jun 19 16:51:59 dist-test-slave-jsw9 krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:51:59.645946 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:59.626482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.1:36187)
0619 16:51:59.632287 (+ 5805us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:59.632362 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:59.633098 (+ 736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:59.634240 (+ 1142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:59.634246 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:59.634651 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:51:59.635022 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:59.635032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.636838 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.636839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:59.638210 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:59.638215 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.639105 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.639107 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:59.639296 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:59.640077 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:51:59.640089 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:51:59.641991 (+ 1902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:51:59.644306 (+ 2315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:59.644314 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:59.644316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:59.644553 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:59.644843 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:59.644844 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:59.644844 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:59.644956 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:59.645391 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:51:59.645395 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:51:59.645515 (+ 120us) client_negotiation.cc:769] Sending connection context
0619 16:51:59.645593 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 16:51:59.645754 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5732,"thread_start_us":136,"threads_started":1}
I20260619 16:51:59.646245 14881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:59.626560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36187 (local address 127.0.98.62:37483)
0619 16:51:59.633371 (+ 6811us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:59.633381 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:59.633441 (+ 60us) server_negotiation.cc:407] Connection header received
0619 16:51:59.633700 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:59.633745 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:59.633823 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:59.633994 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:59.635190 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.636686 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.638324 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.639007 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.639140 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:59.642140 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:59.642147 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:59.642147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:59.642163 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:59.644214 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:59.644633 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:59.644637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:59.644637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:59.644774 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:59.645014 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:59.645014 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:59.645014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:59.645330 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:59.645405 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:59.645668 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 16:51:59.645824 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6741,"thread_start_us":96,"threads_started":1}
I20260619 16:51:59.647003 14878 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37483
I20260619 16:51:59.647114 14878 heartbeater.cc:464] Registering TS with master...
I20260619 16:51:59.647327 14878 heartbeater.cc:511] Master 127.0.98.62:37483 requested a full tablet report, sending...
I20260619 16:51:59.648141 14690 ts_manager.cc:194] Registered new tserver with Master: 0f39a21a9554467bab74fe802df7a1b3 (127.0.98.1:34561)
I20260619 16:51:59.649204 14690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36187
I20260619 16:51:59.650267 14878 heartbeater.cc:503] Master 127.0.98.62:37483 was elected leader, sending a full tablet report...
Jun 19 16:51:59 dist-test-slave-jsw9 krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 16:51:59.657770 14886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:59.648873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54644 (local address 127.0.98.1:34561)
0619 16:51:59.649530 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:59.649535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:59.649545 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:51:59.649581 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:59.649583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:59.649598 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:59.649691 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:59.650112 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.651052 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.651813 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.652135 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.652197 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:59.654676 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:59.654688 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:59.654689 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:59.654698 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:59.656444 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:59.656869 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:59.656873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:59.656873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:59.656933 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:59.657105 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:59.657106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:59.657106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:59.657283 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:59.657363 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:59.657544 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 16:51:59.657653 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":100,"threads_started":1}
I20260619 16:51:59.657783 14885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:59.648794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34561 (local address 127.0.0.1:54644)
0619 16:51:59.649054 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:59.649088 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:59.649154 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:59.649684 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:59.649691 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:59.649977 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:51:59.650034 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:59.650037 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.651121 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.651123 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:59.651727 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:59.651730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.652208 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.652212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:59.652291 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:59.652851 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:51:59.652857 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:51:59.654574 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:51:59.656551 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:59.656559 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:59.656562 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:59.656786 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:59.656992 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:59.656993 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:59.656993 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:59.657053 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:59.657346 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:51:59.657353 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:51:59.657420 (+ 67us) client_negotiation.cc:769] Sending connection context
0619 16:51:59.657493 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 16:51:59.657653 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":117,"threads_started":1}
I20260619 16:51:59.658377 14812 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fba78bc408d4366ad098162d2a85d67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:51:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54644
I20260619 16:51:59.659098 14887 tablet_replica.cc:330] stopping tablet replica
I20260619 16:51:59.659226 14887 raft_consensus.cc:2229] T 9fba78bc408d4366ad098162d2a85d67 P 0f39a21a9554467bab74fe802df7a1b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:51:59.659305 14887 raft_consensus.cc:2258] T 9fba78bc408d4366ad098162d2a85d67 P 0f39a21a9554467bab74fe802df7a1b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:51:59.659696 14887 ts_tablet_manager.cc:1905] T 9fba78bc408d4366ad098162d2a85d67 P 0f39a21a9554467bab74fe802df7a1b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:51:59.662390 14887 ts_tablet_manager.cc:1918] T 9fba78bc408d4366ad098162d2a85d67 P 0f39a21a9554467bab74fe802df7a1b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:51:59.662501 14887 log.cc:1196] T 9fba78bc408d4366ad098162d2a85d67 P 0f39a21a9554467bab74fe802df7a1b3: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/wal/wals/9fba78bc408d4366ad098162d2a85d67
I20260619 16:51:59.662842 14887 ts_tablet_manager.cc:1939] T 9fba78bc408d4366ad098162d2a85d67 P 0f39a21a9554467bab74fe802df7a1b3: Deleting consensus metadata
I20260619 16:51:59.663475 14677 catalog_manager.cc:4841] TS 0f39a21a9554467bab74fe802df7a1b3 (127.0.98.1:34561): tablet 9fba78bc408d4366ad098162d2a85d67 (table test-table [id=ff5a789f4b35414ebb4904358bb468b9]) successfully deleted
W20260619 16:51:59.727835 14882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:51:59.728098 14882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:51:59.728127 14882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:51:59.729385 14882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:51:59.729424 14882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:51:59.729450 14882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:51:59.731045 14882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37581
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37585
--tserver_master_addrs=127.0.98.62:37483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:51:59.731289 14882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:51:59.731516 14882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:51:59.737197 14893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:51:59.737267 14882 server_base.cc:1034] running on GCE node
W20260619 16:51:59.737176 14895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:51:59.737176 14892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:51:59.737669 14882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:51:59.737965 14882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:51:59.739108 14882 hybrid_clock.cc:648] HybridClock initialized: now 1781887919739068 us; error 53 us; skew 500 ppm
Jun 19 16:51:59 dist-test-slave-jsw9 krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:51:59.741817 14882 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:51:59.742362 14882 webserver.cc:460] Webserver started at http://127.0.98.2:37585/ using document root <none> and password file <none>
I20260619 16:51:59.742609 14882 fs_manager.cc:359] Metadata directory not provided
I20260619 16:51:59.742683 14882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:51:59.745189 14882 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:51:59.745920 14902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:59.746114 14882 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:51:59.746238 14882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/wal
uuid: "6a666372e1ed43b9925eb0d2a61e4891"
format_stamp: "Formatted at 2026-06-19 16:51:58 on dist-test-slave-jsw9"
server_key: "83382d978b9301a71f392c15cd6a590c"
server_key_iv: "d3ef65d94487ffc4a731d7550acf2e66"
server_key_version: "encryptionkey@0"
I20260619 16:51:59.746695 14882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:51:59.766834 14882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:51:59.770078 14882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:51:59.770301 14882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:51:59.770583 14882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:51:59.770938 14882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:51:59.770973 14882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:59.771008 14882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:51:59.771027 14882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:59.777874 14882 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37581
I20260619 16:51:59.777988 15015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37581 every 8 connection(s)
I20260619 16:51:59.778360 14882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:51:59.784739 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 14882
I20260619 16:51:59.785112 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40877
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37483
--webserver_port=37129
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:51:59 dist-test-slave-jsw9 krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:51:59.790494 14881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:59.779202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33501 (local address 127.0.98.62:37483)
0619 16:51:59.779278 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:59.779281 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:59.779585 (+ 304us) server_negotiation.cc:407] Connection header received
0619 16:51:59.780344 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:59.780345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:59.780363 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:59.780421 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:59.781537 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.782145 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.783710 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.783941 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.784010 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:59.787035 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:59.787038 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:59.787038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:59.787048 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:59.788795 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:59.789384 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:59.789385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:59.789386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:59.789442 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:59.789732 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:59.789732 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:59.789732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:59.789853 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:59.789903 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:59.790318 (+ 415us) server_negotiation.cc:299] Negotiation successful
0619 16:51:59.790415 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 16:51:59.790805 15018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:59.779190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.2:33501)
0619 16:51:59.779409 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:59.779454 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:59.780211 (+ 757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:59.780505 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:59.780509 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:59.781166 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:51:59.781410 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:59.781416 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.782334 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.782337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:59.783599 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:59.783602 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.784046 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.784047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:59.784189 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:59.784829 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:51:59.784842 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:51:59.786913 (+ 2071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:51:59.788916 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:59.788933 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:59.788937 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:59.789295 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:59.789507 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:59.789508 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:59.789508 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:59.789672 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:59.789909 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:51:59.789915 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:51:59.790123 (+ 208us) client_negotiation.cc:769] Sending connection context
0619 16:51:59.790281 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 16:51:59.790499 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":93,"threads_started":1}
I20260619 16:51:59.791353 15016 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37483
I20260619 16:51:59.791464 15016 heartbeater.cc:464] Registering TS with master...
I20260619 16:51:59.791667 15016 heartbeater.cc:511] Master 127.0.98.62:37483 requested a full tablet report, sending...
I20260619 16:51:59.792105 14690 ts_manager.cc:194] Registered new tserver with Master: 6a666372e1ed43b9925eb0d2a61e4891 (127.0.98.2:37581)
I20260619 16:51:59.792507 14690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33501
W20260619 16:51:59.877727 15019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:51:59.878018 15019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:51:59.878058 15019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:51:59.879494 15019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:51:59.879559 15019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:51:59.879602 15019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:51:59.881325 15019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40877
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37129
--tserver_master_addrs=127.0.98.62:37483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:51:59.881628 15019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:51:59.881883 15019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:51:59.888001 15027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:51:59.888016 15025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:51:59.888109 15019 server_base.cc:1034] running on GCE node
W20260619 16:51:59.888227 15024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:51:59.888674 15019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:51:59.888999 15019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:51:59.890166 15019 hybrid_clock.cc:648] HybridClock initialized: now 1781887919890153 us; error 36 us; skew 500 ppm
Jun 19 16:51:59 dist-test-slave-jsw9 krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:51:59.893205 15019 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:51:59.894094 15019 webserver.cc:460] Webserver started at http://127.0.98.3:37129/ using document root <none> and password file <none>
I20260619 16:51:59.894392 15019 fs_manager.cc:359] Metadata directory not provided
I20260619 16:51:59.894470 15019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:51:59.896121 15019 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:51:59.897047 15034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:59.897246 15019 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:51:59.897360 15019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/wal
uuid: "1e4b5295a038445080cc6ec2e3b5ce1d"
format_stamp: "Formatted at 2026-06-19 16:51:59 on dist-test-slave-jsw9"
server_key: "59c29fdf1cb0dcd2c7299cb792c96ec9"
server_key_iv: "b1d1214912df57469b138d88f8c65dbc"
server_key_version: "encryptionkey@0"
I20260619 16:51:59.897814 15019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:51:59.939788 15019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:51:59.942517 15019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:51:59.942711 15019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:51:59.942991 15019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:51:59.943346 15019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:51:59.943382 15019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:59.943415 15019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:51:59.943434 15019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:51:59.950943 15019 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40877
I20260619 16:51:59.951002 15147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40877 every 8 connection(s)
I20260619 16:51:59.951417 15019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:51:59.952183 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 15019
Jun 19 16:51:59 dist-test-slave-jsw9 krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:51:59.960862 15152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:59.952617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43272 (local address 127.0.98.62:37483)
0619 16:51:59.952735 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:59.952737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:59.952802 (+ 65us) server_negotiation.cc:407] Connection header received
0619 16:51:59.952884 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:59.952885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:59.952899 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:59.952975 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:59.953618 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.954319 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.955164 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.955417 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.955475 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:59.957359 (+ 1884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:59.957361 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:59.957362 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:59.957371 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:59.959629 (+ 2258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:59.960031 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:59.960033 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:59.960034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:59.960105 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:59.960273 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:59.960273 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:59.960274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:59.960405 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:59.960596 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:59.960632 (+ 36us) server_negotiation.cc:299] Negotiation successful
0619 16:51:59.960724 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":51,"threads_started":1}
I20260619 16:51:59.964564 14881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:59.952361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40721 (local address 127.0.98.62:37483)
0619 16:51:59.952422 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:59.952425 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:59.952722 (+ 297us) server_negotiation.cc:407] Connection header received
0619 16:51:59.953676 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:59.953679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:59.953702 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:59.953770 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:51:59.955222 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.956022 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.957853 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.958149 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.958297 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:59.960746 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:51:59.960748 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:51:59.960748 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:51:59.960757 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:51:59.962669 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:59.963262 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:59.963263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:59.963263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:59.963326 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:51:59.963647 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:51:59.963648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:51:59.963648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:51:59.963813 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:51:59.963886 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:59.964391 (+ 505us) server_negotiation.cc:299] Negotiation successful
0619 16:51:59.964483 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 16:51:59.964972 15150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:59.952346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.3:40721)
0619 16:51:59.952564 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:59.952627 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:59.953327 (+ 700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:59.954100 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:59.954105 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:59.954841 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:51:59.955112 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:59.955119 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.956204 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.956206 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:59.957745 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:59.957751 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.958365 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.958366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:59.958563 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:59.959043 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:51:59.959053 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:51:59.960633 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:51:59.962757 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:59.962767 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:59.962770 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:59.963096 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:59.963414 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:51:59.963415 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:51:59.963416 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:51:59.963563 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:51:59.963961 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:51:59.963976 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:51:59.964193 (+ 217us) client_negotiation.cc:769] Sending connection context
0619 16:51:59.964340 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 16:51:59.964720 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":91,"threads_started":1}
I20260619 16:51:59.965772 15148 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37483
I20260619 16:51:59.965914 15148 heartbeater.cc:464] Registering TS with master...
I20260619 16:51:59.966159 15148 heartbeater.cc:511] Master 127.0.98.62:37483 requested a full tablet report, sending...
I20260619 16:51:59.966858 14690 ts_manager.cc:194] Registered new tserver with Master: 1e4b5295a038445080cc6ec2e3b5ce1d (127.0.98.3:40877)
I20260619 16:51:59.967304 14690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40721
I20260619 16:51:59.974260 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:51:59.978127 14881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:59.974754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43280 (local address 127.0.98.62:37483)
0619 16:51:59.974813 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:59.974815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:59.974951 (+ 136us) server_negotiation.cc:407] Connection header received
0619 16:51:59.975024 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:59.975025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:59.975095 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:59.975193 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:51:59.975489 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.976150 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.977188 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.977402 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.977612 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:59.977700 (+ 88us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:51:59.977838 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:51:59.977948 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:59.978016 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 16:51:59.978045 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 16:51:59.978624 14690 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:43280:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:51:59.980644 14690 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:51:59.983402 14812 tablet_service.cc:1467] Processing CreateTablet for tablet e0c7fca2ca914013983c9aa6e4c13461 (DEFAULT_TABLE table=test-table [id=920e97a1267d45bba03cecddb676aa5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:51:59.983681 14812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0c7fca2ca914013983c9aa6e4c13461. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:51:59.985620 14770 tablet_bootstrap.cc:492] T e0c7fca2ca914013983c9aa6e4c13461 P 0f39a21a9554467bab74fe802df7a1b3: Bootstrap starting.
I20260619 16:51:59.986102 14770 tablet_bootstrap.cc:654] T e0c7fca2ca914013983c9aa6e4c13461 P 0f39a21a9554467bab74fe802df7a1b3: Neither blocks nor log segments found. Creating new log.
I20260619 16:51:59.986755 14770 tablet_bootstrap.cc:492] T e0c7fca2ca914013983c9aa6e4c13461 P 0f39a21a9554467bab74fe802df7a1b3: No bootstrap required, opened a new log
I20260619 16:51:59.986809 14770 ts_tablet_manager.cc:1397] T e0c7fca2ca914013983c9aa6e4c13461 P 0f39a21a9554467bab74fe802df7a1b3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 16:51:59.987159 14770 raft_consensus.cc:348] T e0c7fca2ca914013983c9aa6e4c13461 P 0f39a21a9554467bab74fe802df7a1b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a666372e1ed43b9925eb0d2a61e4891" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37581 } } peers { permanent_uuid: "1e4b5295a038445080cc6ec2e3b5ce1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40877 } } peers { permanent_uuid: "0f39a21a9554467bab74fe802df7a1b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34561 } }
I20260619 16:51:59.987231 14770 raft_consensus.cc:374] T e0c7fca2ca914013983c9aa6e4c13461 P 0f39a21a9554467bab74fe802df7a1b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:51:59.987290 14770 raft_consensus.cc:729] T e0c7fca2ca914013983c9aa6e4c13461 P 0f39a21a9554467bab74fe802df7a1b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f39a21a9554467bab74fe802df7a1b3, State: Initialized, Role: FOLLOWER
I20260619 16:51:59.987344 14770 consensus_queue.cc:260] T e0c7fca2ca914013983c9aa6e4c13461 P 0f39a21a9554467bab74fe802df7a1b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a666372e1ed43b9925eb0d2a61e4891" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37581 } } peers { permanent_uuid: "1e4b5295a038445080cc6ec2e3b5ce1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40877 } } peers { permanent_uuid: "0f39a21a9554467bab74fe802df7a1b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34561 } }
I20260619 16:51:59.987450 14770 ts_tablet_manager.cc:1428] T e0c7fca2ca914013983c9aa6e4c13461 P 0f39a21a9554467bab74fe802df7a1b3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 16:51:59.987970 15154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:59.983225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40877 (local address 127.0.0.1:58976)
0619 16:51:59.983368 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:59.983377 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:59.983421 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:59.983946 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:59.983947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:59.983949 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:51:59.983992 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:59.983993 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.985259 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.985260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:59.986660 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:59.986664 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.987605 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.987606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:59.987693 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:59.987724 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 16:51:59.987802 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 16:51:59.987884 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":22,"thread_start_us":108,"threads_started":1}
I20260619 16:51:59.987978 15156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:59.983489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58976 (local address 127.0.98.3:40877)
0619 16:51:59.983776 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:59.983780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:59.983791 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:51:59.983833 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:59.983836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:59.983852 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:59.983941 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:51:59.984059 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.985083 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.986750 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.987527 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.987604 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:59.987677 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:59.987829 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 16:51:59.987882 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":129,"threads_started":1}
I20260619 16:51:59.989006 15082 tablet_service.cc:1467] Processing CreateTablet for tablet e0c7fca2ca914013983c9aa6e4c13461 (DEFAULT_TABLE table=test-table [id=920e97a1267d45bba03cecddb676aa5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:51:59.989367 14885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:59.983120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37581 (local address 127.0.0.1:43818)
0619 16:51:59.983183 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0619 16:51:59.983196 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:51:59.983245 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:51:59.983675 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:51:59.983676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:51:59.983679 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:51:59.983749 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:59.983751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.985085 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.985086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:59.986904 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:51:59.986906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.988837 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.988839 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:51:59.989059 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:59.989070 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 16:51:59.989178 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 16:51:59.989258 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":282}
I20260619 16:51:59.989446 15082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0c7fca2ca914013983c9aa6e4c13461. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:51:59.989742 15155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:51:59.983244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43818 (local address 127.0.98.2:37581)
0619 16:51:59.983477 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 16:51:59.983483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:51:59.983493 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:51:59.983540 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:51:59.983543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:51:59.983560 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:51:59.983820 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:51:59.983898 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.984984 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.987723 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:51:59.988710 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:51:59.988824 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:51:59.988892 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 16:51:59.989433 (+ 541us) server_negotiation.cc:299] Negotiation successful
0619 16:51:59.989524 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":99,"threads_started":1}
I20260619 16:51:59.990477 14950 tablet_service.cc:1467] Processing CreateTablet for tablet e0c7fca2ca914013983c9aa6e4c13461 (DEFAULT_TABLE table=test-table [id=920e97a1267d45bba03cecddb676aa5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:51:59.990803 14950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0c7fca2ca914013983c9aa6e4c13461. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:51:59.994221 15158 tablet_bootstrap.cc:492] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891: Bootstrap starting.
I20260619 16:51:59.994576 15159 tablet_bootstrap.cc:492] T e0c7fca2ca914013983c9aa6e4c13461 P 1e4b5295a038445080cc6ec2e3b5ce1d: Bootstrap starting.
I20260619 16:51:59.995102 15158 tablet_bootstrap.cc:654] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891: Neither blocks nor log segments found. Creating new log.
I20260619 16:51:59.995285 15159 tablet_bootstrap.cc:654] T e0c7fca2ca914013983c9aa6e4c13461 P 1e4b5295a038445080cc6ec2e3b5ce1d: Neither blocks nor log segments found. Creating new log.
I20260619 16:51:59.995811 15159 log.cc:824] T e0c7fca2ca914013983c9aa6e4c13461 P 1e4b5295a038445080cc6ec2e3b5ce1d: Log is configured to *not* fsync() on all Append() calls
I20260619 16:51:59.996006 15158 log.cc:824] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891: Log is configured to *not* fsync() on all Append() calls
I20260619 16:51:59.996901 15159 tablet_bootstrap.cc:492] T e0c7fca2ca914013983c9aa6e4c13461 P 1e4b5295a038445080cc6ec2e3b5ce1d: No bootstrap required, opened a new log
I20260619 16:51:59.996896 15158 tablet_bootstrap.cc:492] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891: No bootstrap required, opened a new log
I20260619 16:51:59.996980 15158 ts_tablet_manager.cc:1397] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:51:59.996980 15159 ts_tablet_manager.cc:1397] T e0c7fca2ca914013983c9aa6e4c13461 P 1e4b5295a038445080cc6ec2e3b5ce1d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:51:59.999359 15159 raft_consensus.cc:348] T e0c7fca2ca914013983c9aa6e4c13461 P 1e4b5295a038445080cc6ec2e3b5ce1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a666372e1ed43b9925eb0d2a61e4891" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37581 } } peers { permanent_uuid: "1e4b5295a038445080cc6ec2e3b5ce1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40877 } } peers { permanent_uuid: "0f39a21a9554467bab74fe802df7a1b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34561 } }
I20260619 16:51:59.999359 15158 raft_consensus.cc:348] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a666372e1ed43b9925eb0d2a61e4891" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37581 } } peers { permanent_uuid: "1e4b5295a038445080cc6ec2e3b5ce1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40877 } } peers { permanent_uuid: "0f39a21a9554467bab74fe802df7a1b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34561 } }
I20260619 16:51:59.999614 15158 raft_consensus.cc:374] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:51:59.999614 15159 raft_consensus.cc:374] T e0c7fca2ca914013983c9aa6e4c13461 P 1e4b5295a038445080cc6ec2e3b5ce1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:51:59.999651 15158 raft_consensus.cc:729] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a666372e1ed43b9925eb0d2a61e4891, State: Initialized, Role: FOLLOWER
I20260619 16:51:59.999651 15159 raft_consensus.cc:729] T e0c7fca2ca914013983c9aa6e4c13461 P 1e4b5295a038445080cc6ec2e3b5ce1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e4b5295a038445080cc6ec2e3b5ce1d, State: Initialized, Role: FOLLOWER
I20260619 16:51:59.999799 15159 consensus_queue.cc:260] T e0c7fca2ca914013983c9aa6e4c13461 P 1e4b5295a038445080cc6ec2e3b5ce1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a666372e1ed43b9925eb0d2a61e4891" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37581 } } peers { permanent_uuid: "1e4b5295a038445080cc6ec2e3b5ce1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40877 } } peers { permanent_uuid: "0f39a21a9554467bab74fe802df7a1b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34561 } }
I20260619 16:51:59.999805 15158 consensus_queue.cc:260] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a666372e1ed43b9925eb0d2a61e4891" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37581 } } peers { permanent_uuid: "1e4b5295a038445080cc6ec2e3b5ce1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40877 } } peers { permanent_uuid: "0f39a21a9554467bab74fe802df7a1b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34561 } }
I20260619 16:52:00.000177 15158 ts_tablet_manager.cc:1428] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260619 16:52:00.000177 15159 ts_tablet_manager.cc:1428] T e0c7fca2ca914013983c9aa6e4c13461 P 1e4b5295a038445080cc6ec2e3b5ce1d: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260619 16:52:00.000226 15148 heartbeater.cc:503] Master 127.0.98.62:37483 was elected leader, sending a full tablet report...
I20260619 16:52:00.000319 15016 heartbeater.cc:503] Master 127.0.98.62:37483 was elected leader, sending a full tablet report...
W20260619 16:52:00.029063 15017 tablet.cc:2369] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:52:00.110584 15162 raft_consensus.cc:482] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:52:00.110782 15162 raft_consensus.cc:504] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a666372e1ed43b9925eb0d2a61e4891" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37581 } } peers { permanent_uuid: "1e4b5295a038445080cc6ec2e3b5ce1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40877 } } peers { permanent_uuid: "0f39a21a9554467bab74fe802df7a1b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34561 } }
I20260619 16:52:00.111294 15162 leader_election.cc:288] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e4b5295a038445080cc6ec2e3b5ce1d (127.0.98.3:40877), 0f39a21a9554467bab74fe802df7a1b3 (127.0.98.1:34561)
I20260619 16:52:00.115064 14886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:00.111737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42355 (local address 127.0.98.1:34561)
0619 16:52:00.111841 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:00.111843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:00.111854 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:52:00.111910 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:00.111912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:00.111946 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:00.112015 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:52:00.112227 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:00.113087 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:00.114240 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:00.114712 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:00.114786 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:00.114898 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:00.114927 (+ 29us) server_negotiation.cc:299] Negotiation successful
0619 16:52:00.114969 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 16:52:00.115063 15156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:00.111649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45633 (local address 127.0.98.3:40877)
0619 16:52:00.111789 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:00.111792 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:00.111983 (+ 191us) server_negotiation.cc:407] Connection header received
0619 16:52:00.112050 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:00.112052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:00.112066 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:00.112143 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:52:00.112313 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:00.113084 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:00.114248 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:00.114678 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:00.114740 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:00.114751 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:00.114924 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 16:52:00.114971 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260619 16:52:00.115091 15018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:00.111563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34561 (local address 127.0.98.2:42355)
0619 16:52:00.111695 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:00.111709 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:00.111808 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:00.112040 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:00.112041 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:00.112042 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:52:00.112129 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:00.112132 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:00.113198 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:00.113200 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:00.114112 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:00.114116 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:00.114799 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:00.114799 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:00.114868 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:00.114871 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:52:00.114907 (+ 36us) client_negotiation.cc:240] Negotiation successful
0619 16:52:00.114960 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":64}
I20260619 16:52:00.115119 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:00.111573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40877 (local address 127.0.98.2:45633)
0619 16:52:00.111884 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:00.111894 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:00.111969 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:00.112146 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:00.112147 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:00.112149 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:52:00.112215 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:00.112216 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:00.113198 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:00.113199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:00.114102 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:00.114105 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:00.114767 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:00.114768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:00.114845 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:00.114850 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:52:00.114903 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 16:52:00.114958 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":4,"thread_start_us":171,"threads_started":1}
I20260619 16:52:00.115528 14831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c7fca2ca914013983c9aa6e4c13461" candidate_uuid: "6a666372e1ed43b9925eb0d2a61e4891" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f39a21a9554467bab74fe802df7a1b3" is_pre_election: true
I20260619 16:52:00.115593 15102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c7fca2ca914013983c9aa6e4c13461" candidate_uuid: "6a666372e1ed43b9925eb0d2a61e4891" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e4b5295a038445080cc6ec2e3b5ce1d" is_pre_election: true
I20260619 16:52:00.115751 15102 raft_consensus.cc:2454] T e0c7fca2ca914013983c9aa6e4c13461 P 1e4b5295a038445080cc6ec2e3b5ce1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a666372e1ed43b9925eb0d2a61e4891 in term 0.
I20260619 16:52:00.115752 14831 raft_consensus.cc:2454] T e0c7fca2ca914013983c9aa6e4c13461 P 0f39a21a9554467bab74fe802df7a1b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a666372e1ed43b9925eb0d2a61e4891 in term 0.
I20260619 16:52:00.116017 14906 leader_election.cc:302] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f39a21a9554467bab74fe802df7a1b3, 6a666372e1ed43b9925eb0d2a61e4891; no voters:
I20260619 16:52:00.116255 15162 raft_consensus.cc:2781] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:52:00.116300 15162 raft_consensus.cc:482] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:52:00.116315 15162 raft_consensus.cc:3037] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:52:00.117153 15162 raft_consensus.cc:504] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a666372e1ed43b9925eb0d2a61e4891" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37581 } } peers { permanent_uuid: "1e4b5295a038445080cc6ec2e3b5ce1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40877 } } peers { permanent_uuid: "0f39a21a9554467bab74fe802df7a1b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34561 } }
I20260619 16:52:00.117316 15162 leader_election.cc:288] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 [CANDIDATE]: Term 1 election: Requested vote from peers 1e4b5295a038445080cc6ec2e3b5ce1d (127.0.98.3:40877), 0f39a21a9554467bab74fe802df7a1b3 (127.0.98.1:34561)
I20260619 16:52:00.117551 15102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c7fca2ca914013983c9aa6e4c13461" candidate_uuid: "6a666372e1ed43b9925eb0d2a61e4891" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e4b5295a038445080cc6ec2e3b5ce1d"
I20260619 16:52:00.117614 14831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c7fca2ca914013983c9aa6e4c13461" candidate_uuid: "6a666372e1ed43b9925eb0d2a61e4891" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f39a21a9554467bab74fe802df7a1b3"
I20260619 16:52:00.117654 15102 raft_consensus.cc:3037] T e0c7fca2ca914013983c9aa6e4c13461 P 1e4b5295a038445080cc6ec2e3b5ce1d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:52:00.117717 14831 raft_consensus.cc:3037] T e0c7fca2ca914013983c9aa6e4c13461 P 0f39a21a9554467bab74fe802df7a1b3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:52:00.118573 15102 raft_consensus.cc:2454] T e0c7fca2ca914013983c9aa6e4c13461 P 1e4b5295a038445080cc6ec2e3b5ce1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a666372e1ed43b9925eb0d2a61e4891 in term 1.
I20260619 16:52:00.118608 14831 raft_consensus.cc:2454] T e0c7fca2ca914013983c9aa6e4c13461 P 0f39a21a9554467bab74fe802df7a1b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a666372e1ed43b9925eb0d2a61e4891 in term 1.
I20260619 16:52:00.118790 14905 leader_election.cc:302] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e4b5295a038445080cc6ec2e3b5ce1d, 6a666372e1ed43b9925eb0d2a61e4891; no voters:
I20260619 16:52:00.118891 15162 raft_consensus.cc:2781] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:52:00.119083 15162 raft_consensus.cc:686] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 [term 1 LEADER]: Becoming Leader. State: Replica: 6a666372e1ed43b9925eb0d2a61e4891, State: Running, Role: LEADER
I20260619 16:52:00.119186 15162 consensus_queue.cc:237] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a666372e1ed43b9925eb0d2a61e4891" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37581 } } peers { permanent_uuid: "1e4b5295a038445080cc6ec2e3b5ce1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40877 } } peers { permanent_uuid: "0f39a21a9554467bab74fe802df7a1b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34561 } }
I20260619 16:52:00.120265 14689 catalog_manager.cc:5495] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a666372e1ed43b9925eb0d2a61e4891 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6a666372e1ed43b9925eb0d2a61e4891" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a666372e1ed43b9925eb0d2a61e4891" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e4b5295a038445080cc6ec2e3b5ce1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f39a21a9554467bab74fe802df7a1b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34561 } health_report { overall_health: UNKNOWN } } }
W20260619 16:52:00.130848 14879 tablet.cc:2369] T e0c7fca2ca914013983c9aa6e4c13461 P 0f39a21a9554467bab74fe802df7a1b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:52:00.140985 15155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:00.138590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43820 (local address 127.0.98.2:37581)
0619 16:52:00.138659 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:00.138661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:00.138669 (+ 8us) server_negotiation.cc:407] Connection header received
0619 16:52:00.138718 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:00.138719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:00.138744 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:00.138815 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:52:00.138945 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:00.139476 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:00.140224 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:00.140426 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:00.140483 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:00.140625 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:52:00.140755 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:52:00.140844 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:00.140860 (+ 16us) server_negotiation.cc:299] Negotiation successful
0619 16:52:00.140886 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 16:52:00.143294 15102 raft_consensus.cc:1261] T e0c7fca2ca914013983c9aa6e4c13461 P 1e4b5295a038445080cc6ec2e3b5ce1d [term 1 FOLLOWER]: Refusing update from remote peer 6a666372e1ed43b9925eb0d2a61e4891: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:52:00.143297 14831 raft_consensus.cc:1261] T e0c7fca2ca914013983c9aa6e4c13461 P 0f39a21a9554467bab74fe802df7a1b3 [term 1 FOLLOWER]: Refusing update from remote peer 6a666372e1ed43b9925eb0d2a61e4891: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:52:00.143658 15162 consensus_queue.cc:1035] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f39a21a9554467bab74fe802df7a1b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:52:00.143833 15165 consensus_queue.cc:1035] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e4b5295a038445080cc6ec2e3b5ce1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:52:00.146848 15170 mvcc.cc:205] Tried to move back new op lower bound from 7298612920903282688 to 7298612920808886272. Current Snapshot: MvccSnapshot[applied={T|T < 7298612920903282688}]
I20260619 16:52:00.146957 15169 mvcc.cc:205] Tried to move back new op lower bound from 7298612920903282688 to 7298612920808886272. Current Snapshot: MvccSnapshot[applied={T|T < 7298612920903282688}]
I20260619 16:52:00.148762 14689 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:43280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:52:00.148864 14689 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:43280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:52:00.149658 14689 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3: Sending DeleteTablet for 3 replicas of tablet e0c7fca2ca914013983c9aa6e4c13461
I20260619 16:52:00.149961 14950 tablet_service.cc:1514] Processing DeleteTablet for tablet e0c7fca2ca914013983c9aa6e4c13461 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:52:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43818
I20260619 16:52:00.150123 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 14728
I20260619 16:52:00.150177 14812 tablet_service.cc:1514] Processing DeleteTablet for tablet e0c7fca2ca914013983c9aa6e4c13461 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:52:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54644
I20260619 16:52:00.150297 14887 tablet_replica.cc:330] T e0c7fca2ca914013983c9aa6e4c13461 P 0f39a21a9554467bab74fe802df7a1b3: stopping tablet replica
I20260619 16:52:00.150343 14887 raft_consensus.cc:2229] T e0c7fca2ca914013983c9aa6e4c13461 P 0f39a21a9554467bab74fe802df7a1b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:52:00.150393 14887 raft_consensus.cc:2258] T e0c7fca2ca914013983c9aa6e4c13461 P 0f39a21a9554467bab74fe802df7a1b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:52:00.152892 15082 tablet_service.cc:1514] Processing DeleteTablet for tablet e0c7fca2ca914013983c9aa6e4c13461 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:52:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58976
I20260619 16:52:00.153187 15179 tablet_replica.cc:330] T e0c7fca2ca914013983c9aa6e4c13461 P 1e4b5295a038445080cc6ec2e3b5ce1d: stopping tablet replica
I20260619 16:52:00.153337 15179 raft_consensus.cc:2229] T e0c7fca2ca914013983c9aa6e4c13461 P 1e4b5295a038445080cc6ec2e3b5ce1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:52:00.153499 15179 raft_consensus.cc:2258] T e0c7fca2ca914013983c9aa6e4c13461 P 1e4b5295a038445080cc6ec2e3b5ce1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:52:00.154071 15179 ts_tablet_manager.cc:1905] T e0c7fca2ca914013983c9aa6e4c13461 P 1e4b5295a038445080cc6ec2e3b5ce1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:52:00.156054 15179 ts_tablet_manager.cc:1918] T e0c7fca2ca914013983c9aa6e4c13461 P 1e4b5295a038445080cc6ec2e3b5ce1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:52:00.156144 15179 log.cc:1196] T e0c7fca2ca914013983c9aa6e4c13461 P 1e4b5295a038445080cc6ec2e3b5ce1d: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/wal/wals/e0c7fca2ca914013983c9aa6e4c13461
I20260619 16:52:00.156433 15179 ts_tablet_manager.cc:1939] T e0c7fca2ca914013983c9aa6e4c13461 P 1e4b5295a038445080cc6ec2e3b5ce1d: Deleting consensus metadata
I20260619 16:52:00.157099 14676 catalog_manager.cc:4841] TS 1e4b5295a038445080cc6ec2e3b5ce1d (127.0.98.3:40877): tablet e0c7fca2ca914013983c9aa6e4c13461 (table test-table [id=920e97a1267d45bba03cecddb676aa5d]) successfully deleted
W20260619 16:52:00.157729 14677 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34561 (error 108)
I20260619 16:52:00.157843 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 14882
I20260619 16:52:00.157858 15178 tablet_replica.cc:330] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891: stopping tablet replica
I20260619 16:52:00.162875 14885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:00.162718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34561 (local address 127.0.0.1:54656)
0619 16:52:00.162780 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:00.162828 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 16:52:00.163074 14677 catalog_manager.cc:4584] TS 0f39a21a9554467bab74fe802df7a1b3 (127.0.98.1:34561): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e0c7fca2ca914013983c9aa6e4c13461: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34561: connect: Connection refused (error 111)
I20260619 16:52:00.163729 14885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:00.163621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37581 (local address 127.0.0.1:43822)
0619 16:52:00.163670 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:00.163694 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:37581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 16:52:00.163911 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 15019
I20260619 16:52:00.168727 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 14657
I20260619 16:52:00.174324 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37483
--webserver_interface=127.0.98.62
--webserver_port=46495
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37483
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:52:00.276530 15181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:00.276850 15181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:00.276880 15181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:00.278393 15181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:52:00.278470 15181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:00.278486 15181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:00.278498 15181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:52:00.278510 15181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:52:00.280725 15181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:52:00.281111 15181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:00.281404 15181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:00.287525 15187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:00.287770 15181 server_base.cc:1034] running on GCE node
W20260619 16:52:00.287514 15186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:00.287514 15189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:00.288385 15181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:00.288767 15181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:00.289986 15181 hybrid_clock.cc:648] HybridClock initialized: now 1781887920289960 us; error 67 us; skew 500 ppm
Jun 19 16:52:00 dist-test-slave-jsw9 krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:00.293387 15181 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:52:00.294100 15181 webserver.cc:460] Webserver started at http://127.0.98.62:46495/ using document root <none> and password file <none>
I20260619 16:52:00.294430 15181 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:00.294497 15181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:00.296276 15181 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 16:52:00.297622 15196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:00.297960 15181 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:52:00.298079 15181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "906da5240bf9440384008a0ac35da4d3"
format_stamp: "Formatted at 2026-06-19 16:51:58 on dist-test-slave-jsw9"
server_key: "1b5a3fa26ccee2827a750b4d96bc44cd"
server_key_iv: "e4ce5376931a2b866f0b03715ed9556f"
server_key_version: "encryptionkey@0"
I20260619 16:52:00.298615 15181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:00.317931 15181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:00.321116 15181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:00.321317 15181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:00.327164 15181 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37483
I20260619 16:52:00.327263 15248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37483 every 8 connection(s)
I20260619 16:52:00.327687 15181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:52:00.329706 15249 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:52:00.330588 15249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3: Bootstrap starting.
I20260619 16:52:00.331673 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 15181
I20260619 16:52:00.332023 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34561
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37483
--webserver_port=36533
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:52:00.342566 15249 log.cc:824] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3: Log is configured to *not* fsync() on all Append() calls
I20260619 16:52:00.345160 15249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:52:00.345506 15249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3: Bootstrap complete.
I20260619 16:52:00.348096 15249 raft_consensus.cc:348] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "906da5240bf9440384008a0ac35da4d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37483 } }
I20260619 16:52:00.348402 15249 raft_consensus.cc:729] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 906da5240bf9440384008a0ac35da4d3, State: Initialized, Role: FOLLOWER
I20260619 16:52:00.348568 15249 consensus_queue.cc:260] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "906da5240bf9440384008a0ac35da4d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37483 } }
I20260619 16:52:00.349066 15249 raft_consensus.cc:388] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:52:00.349159 15249 raft_consensus.cc:482] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:52:00.349232 15249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [term 2 FOLLOWER]: Advancing to term 3
I20260619 16:52:00.351166 15249 raft_consensus.cc:504] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "906da5240bf9440384008a0ac35da4d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37483 } }
I20260619 16:52:00.351306 15249 leader_election.cc:302] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 906da5240bf9440384008a0ac35da4d3; no voters:
I20260619 16:52:00.351718 15249 leader_election.cc:288] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 16:52:00.351825 15253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 16:52:00.352002 15253 raft_consensus.cc:686] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [term 3 LEADER]: Becoming Leader. State: Replica: 906da5240bf9440384008a0ac35da4d3, State: Running, Role: LEADER
I20260619 16:52:00.352144 15249 sys_catalog.cc:564] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:52:00.352159 15253 consensus_queue.cc:237] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "906da5240bf9440384008a0ac35da4d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37483 } }
I20260619 16:52:00.352588 15253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "906da5240bf9440384008a0ac35da4d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "906da5240bf9440384008a0ac35da4d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37483 } } }
I20260619 16:52:00.352792 15253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [sys.catalog]: This master's current role is: LEADER
I20260619 16:52:00.352990 15253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 906da5240bf9440384008a0ac35da4d3. Latest consensus state: current_term: 3 leader_uuid: "906da5240bf9440384008a0ac35da4d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "906da5240bf9440384008a0ac35da4d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37483 } } }
I20260619 16:52:00.353116 15253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 906da5240bf9440384008a0ac35da4d3 [sys.catalog]: This master's current role is: LEADER
I20260619 16:52:00.353302 15262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:52:00.354472 15262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:52:00.354674 15262 catalog_manager.cc:1259] Loaded cluster ID: fd60ec7e79c6475791220ba6d70e7aae
I20260619 16:52:00.354928 15262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 16:52:00.355204 15262 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6ea29069ad google::LogMessage::Fail() at ??:0
@ 0x7f6ea290988a google::LogMessage::SendToLog() at ??:0
@ 0x7f6ea29064de google::LogMessage::Flush() at ??:0
@ 0x7f6ea2907a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6eaa82b0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f6eaa8348b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6ea47d40d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6ea47c9954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6ea562d6db start_thread at ??:0
@ 0x7f6ea140461f clone at ??:0
W20260619 16:52:00.444697 15251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:00.444952 15251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:00.444988 15251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:00.446358 15251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:00.446406 15251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:00.446445 15251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:52:00.448115 15251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34561
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36533
--tserver_master_addrs=127.0.98.62:37483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:52:00.448433 15251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:00.448680 15251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:00.454370 15285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:00.454432 15286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:00.454349 15288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:00.454530 15251 server_base.cc:1034] running on GCE node
I20260619 16:52:00.454763 15251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:00.455051 15251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:00.456239 15251 hybrid_clock.cc:648] HybridClock initialized: now 1781887920456222 us; error 33 us; skew 500 ppm
Jun 19 16:52:00 dist-test-slave-jsw9 krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:00.459246 15251 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:52:00.459911 15251 webserver.cc:460] Webserver started at http://127.0.98.1:36533/ using document root <none> and password file <none>
I20260619 16:52:00.460217 15251 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:00.460279 15251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:00.462249 15251 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 16:52:00.463114 15295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:00.463305 15251 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:52:00.463415 15251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/wal
uuid: "0f39a21a9554467bab74fe802df7a1b3"
format_stamp: "Formatted at 2026-06-19 16:51:58 on dist-test-slave-jsw9"
server_key: "5ee1a4fbc34fe1be61f11e088a0c78f6"
server_key_iv: "9897c7ae7c8ad362dbc3bbe0f67acb08"
server_key_version: "encryptionkey@0"
I20260619 16:52:00.463928 15251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:00.484963 15251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:00.487722 15251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:00.487864 15251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:00.488204 15251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:52:00.488886 15302 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 16:52:00.491070 15251 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 16:52:00.491147 15251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 16:52:00.491636 15251 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 16:52:00.492604 15251 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 16:52:00.492656 15251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 16:52:00.492743 15302 tablet_bootstrap.cc:492] T e0c7fca2ca914013983c9aa6e4c13461 P 0f39a21a9554467bab74fe802df7a1b3: Bootstrap starting.
I20260619 16:52:00.502367 15251 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34561
I20260619 16:52:00.502496 15409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34561 every 8 connection(s)
I20260619 16:52:00.502947 15251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:52:00.506220 15302 log.cc:824] T e0c7fca2ca914013983c9aa6e4c13461 P 0f39a21a9554467bab74fe802df7a1b3: Log is configured to *not* fsync() on all Append() calls
I20260619 16:52:00.508952 15302 tablet_bootstrap.cc:492] T e0c7fca2ca914013983c9aa6e4c13461 P 0f39a21a9554467bab74fe802df7a1b3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:52:00.509298 15302 tablet_bootstrap.cc:492] T e0c7fca2ca914013983c9aa6e4c13461 P 0f39a21a9554467bab74fe802df7a1b3: Bootstrap complete.
I20260619 16:52:00.509326 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 15251
I20260619 16:52:00.509832 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37581
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37483
--webserver_port=37585
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:52:00.510277 15302 ts_tablet_manager.cc:1397] T e0c7fca2ca914013983c9aa6e4c13461 P 0f39a21a9554467bab74fe802df7a1b3: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.010s
I20260619 16:52:00.511686 15302 raft_consensus.cc:348] T e0c7fca2ca914013983c9aa6e4c13461 P 0f39a21a9554467bab74fe802df7a1b3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a666372e1ed43b9925eb0d2a61e4891" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37581 } } peers { permanent_uuid: "1e4b5295a038445080cc6ec2e3b5ce1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40877 } } peers { permanent_uuid: "0f39a21a9554467bab74fe802df7a1b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34561 } }
I20260619 16:52:00.512392 15302 raft_consensus.cc:729] T e0c7fca2ca914013983c9aa6e4c13461 P 0f39a21a9554467bab74fe802df7a1b3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f39a21a9554467bab74fe802df7a1b3, State: Initialized, Role: FOLLOWER
I20260619 16:52:00.512627 15302 consensus_queue.cc:260] T e0c7fca2ca914013983c9aa6e4c13461 P 0f39a21a9554467bab74fe802df7a1b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a666372e1ed43b9925eb0d2a61e4891" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37581 } } peers { permanent_uuid: "1e4b5295a038445080cc6ec2e3b5ce1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40877 } } peers { permanent_uuid: "0f39a21a9554467bab74fe802df7a1b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34561 } }
I20260619 16:52:00.512977 15302 ts_tablet_manager.cc:1428] T e0c7fca2ca914013983c9aa6e4c13461 P 0f39a21a9554467bab74fe802df7a1b3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
W20260619 16:52:00.609529 15414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:00.609812 15414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:00.609841 15414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:00.611290 15414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:00.611366 15414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:00.611485 15414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:52:00.613498 15414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37581
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37585
--tserver_master_addrs=127.0.98.62:37483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:52:00.613817 15414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:00.614089 15414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:00.620051 15421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:00.620051 15423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:00.620177 15420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:00.620868 15414 server_base.cc:1034] running on GCE node
I20260619 16:52:00.621187 15414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:00.621524 15414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:00.622720 15414 hybrid_clock.cc:648] HybridClock initialized: now 1781887920622664 us; error 73 us; skew 500 ppm
Jun 19 16:52:00 dist-test-slave-jsw9 krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:00.626035 15414 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
W20260619 16:52:00.626430 15412 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:52:00.503856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.1:55211)
0619 16:52:00.504134 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:00.504219 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:00.505138 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:00.626152 (+121014us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":131,"threads_started":1}
W20260619 16:52:00.626662 15410 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37483 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37483: Client connection negotiation failed: client connection to 127.0.98.62:37483: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 16:52:00.626755 15414 webserver.cc:460] Webserver started at http://127.0.98.2:37585/ using document root <none> and password file <none>
I20260619 16:52:00.627030 15412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:00.626890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.1:60275)
0619 16:52:00.626940 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:00.626997 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260619 16:52:00.627048 15414 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:00.627107 15414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:00.627336 15412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:00.627242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.1:48049)
0619 16:52:00.627286 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:00.627314 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260619 16:52:00.627647 15410 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 16:52:00.628823 15414 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 16:52:00.629757 15430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:00.630010 15414 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:52:00.630115 15414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/wal
uuid: "6a666372e1ed43b9925eb0d2a61e4891"
format_stamp: "Formatted at 2026-06-19 16:51:58 on dist-test-slave-jsw9"
server_key: "83382d978b9301a71f392c15cd6a590c"
server_key_iv: "d3ef65d94487ffc4a731d7550acf2e66"
server_key_version: "encryptionkey@0"
I20260619 16:52:00.630628 15414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:00.647488 15414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:00.649734 15414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:00.649893 15414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:00.650169 15414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:52:00.650693 15437 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 16:52:00.652443 15414 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 16:52:00.652540 15414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 16:52:00.652587 15414 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 16:52:00.653376 15414 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 16:52:00.653419 15414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 16:52:00.653527 15437 tablet_bootstrap.cc:492] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891: Bootstrap starting.
I20260619 16:52:00.662031 15414 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37581
I20260619 16:52:00.662333 15544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37581 every 8 connection(s)
I20260619 16:52:00.662604 15414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:52:00.664043 15547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:00.663545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.2:52653)
0619 16:52:00.663789 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:00.663880 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":121,"threads_started":1}
W20260619 16:52:00.664258 15545 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37483 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37483: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
I20260619 16:52:00.664608 15547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:00.664483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.2:40089)
0619 16:52:00.664550 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:00.664584 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 16:52:00.664906 15547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:00.664829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.2:45783)
0619 16:52:00.664865 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:00.664889 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 16:52:00.665064 15545 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 16:52:00.667090 15437 log.cc:824] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891: Log is configured to *not* fsync() on all Append() calls
I20260619 16:52:00.667537 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 15414
I20260619 16:52:00.667820 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40877
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37483
--webserver_port=37129
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:52:00.670325 15437 tablet_bootstrap.cc:492] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:52:00.670614 15437 tablet_bootstrap.cc:492] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891: Bootstrap complete.
I20260619 16:52:00.671526 15437 ts_tablet_manager.cc:1397] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.012s
I20260619 16:52:00.672931 15437 raft_consensus.cc:348] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a666372e1ed43b9925eb0d2a61e4891" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37581 } } peers { permanent_uuid: "1e4b5295a038445080cc6ec2e3b5ce1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40877 } } peers { permanent_uuid: "0f39a21a9554467bab74fe802df7a1b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34561 } }
I20260619 16:52:00.673266 15437 raft_consensus.cc:729] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a666372e1ed43b9925eb0d2a61e4891, State: Initialized, Role: FOLLOWER
I20260619 16:52:00.673408 15437 consensus_queue.cc:260] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a666372e1ed43b9925eb0d2a61e4891" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37581 } } peers { permanent_uuid: "1e4b5295a038445080cc6ec2e3b5ce1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40877 } } peers { permanent_uuid: "0f39a21a9554467bab74fe802df7a1b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34561 } }
I20260619 16:52:00.673695 15437 ts_tablet_manager.cc:1428] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260619 16:52:00.674042 15547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:00.673896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.2:38705)
0619 16:52:00.673955 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:00.674001 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 16:52:00.674391 15547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:00.674301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.2:54099)
0619 16:52:00.674342 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:00.674369 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 16:52:00.762830 15549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:00.763115 15549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:00.763149 15549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:00.764716 15549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:00.764792 15549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:00.764827 15549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:52:00.766638 15549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40877
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37129
--tserver_master_addrs=127.0.98.62:37483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:52:00.766968 15549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:00.767236 15549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:00.773041 15556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:00.773144 15549 server_base.cc:1034] running on GCE node
W20260619 16:52:00.773224 15555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:00.773062 15558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:00.773516 15549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:00.773797 15549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:00.774986 15549 hybrid_clock.cc:648] HybridClock initialized: now 1781887920774966 us; error 35 us; skew 500 ppm
Jun 19 16:52:00 dist-test-slave-jsw9 krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:00.777844 15549 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:52:00.778366 15549 webserver.cc:460] Webserver started at http://127.0.98.3:37129/ using document root <none> and password file <none>
I20260619 16:52:00.778570 15549 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:00.778612 15549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:00.780047 15549 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:52:00.780756 15565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:00.780992 15549 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:52:00.781078 15549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/wal
uuid: "1e4b5295a038445080cc6ec2e3b5ce1d"
format_stamp: "Formatted at 2026-06-19 16:51:59 on dist-test-slave-jsw9"
server_key: "59c29fdf1cb0dcd2c7299cb792c96ec9"
server_key_iv: "b1d1214912df57469b138d88f8c65dbc"
server_key_version: "encryptionkey@0"
I20260619 16:52:00.781518 15549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:00.800917 15549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:00.803395 15549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:00.803583 15549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:00.803889 15549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:52:00.804296 15549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:52:00.804345 15549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:00.804388 15549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:52:00.804411 15549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:00.811273 15549 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40877
I20260619 16:52:00.811326 15678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40877 every 8 connection(s)
I20260619 16:52:00.811748 15549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:52:00.813036 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:00.812545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.3:55345)
0619 16:52:00.812751 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:00.812856 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":77,"threads_started":1}
W20260619 16:52:00.813488 15679 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37483 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37483: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
I20260619 16:52:00.813767 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:00.813673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.3:45877)
0619 16:52:00.813714 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:00.813744 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 16:52:00.814064 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:00.813998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.3:57867)
0619 16:52:00.814030 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:00.814050 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260619 16:52:00.814149 15679 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 16:52:00.815832 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 15549
I20260619 16:52:01.628437 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:01.628106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.1:49619)
0619 16:52:01.628286 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:01.628350 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":78,"threads_started":1}
I20260619 16:52:01.675335 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:01.675001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.2:37031)
0619 16:52:01.675183 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:01.675249 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":86,"threads_started":1}
I20260619 16:52:01.814975 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:01.814621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.3:33055)
0619 16:52:01.814838 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:01.814891 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":112,"threads_started":1}
I20260619 16:52:02.000746 15686 raft_consensus.cc:482] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:52:02.000937 15686 raft_consensus.cc:504] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a666372e1ed43b9925eb0d2a61e4891" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37581 } } peers { permanent_uuid: "1e4b5295a038445080cc6ec2e3b5ce1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40877 } } peers { permanent_uuid: "0f39a21a9554467bab74fe802df7a1b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34561 } }
I20260619 16:52:02.001365 15686 leader_election.cc:288] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e4b5295a038445080cc6ec2e3b5ce1d (127.0.98.3:40877), 0f39a21a9554467bab74fe802df7a1b3 (127.0.98.1:34561)
Jun 19 16:52:02 dist-test-slave-jsw9 krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Jun 19 16:52:02 dist-test-slave-jsw9 krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 16:52:02.019769 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:02.001831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59225 (local address 127.0.98.1:34561)
0619 16:52:02.002005 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:02.002011 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:02.002027 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:52:02.002227 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:02.002230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:02.002263 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:02.002352 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:02.003157 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:02.004405 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:02.005244 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:02.005671 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:02.005785 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:02.015525 (+ 9740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:02.015530 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:02.015530 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:02.015544 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:02.017942 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:02.018424 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:02.018430 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:02.018430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:02.018534 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:02.018778 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:02.018779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:02.018780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:02.019083 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:02.019167 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:02.019497 (+ 330us) server_negotiation.cc:299] Negotiation successful
0619 16:52:02.019637 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":59,"threads_started":1}
I20260619 16:52:02.019769 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:02.001638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34561 (local address 127.0.98.2:59225)
0619 16:52:02.001894 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:02.001909 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:02.002056 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:02.002409 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:02.002415 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:02.002798 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:52:02.003033 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:02.003040 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:02.004545 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:02.004546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:02.005144 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:02.005147 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:02.005763 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:02.005764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:02.005884 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:02.012622 (+ 6738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:52:02.012641 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:52:02.015373 (+ 2732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:52:02.018041 (+ 2668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:02.018042 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:02.018043 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:02.018339 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:02.018596 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:02.018597 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:02.018598 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:02.018721 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:02.019154 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:52:02.019162 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:52:02.019309 (+ 147us) client_negotiation.cc:769] Sending connection context
0619 16:52:02.019418 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 16:52:02.019637 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":75,"thread_start_us":81,"threads_started":1}
I20260619 16:52:02.020221 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:02.001638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40877 (local address 127.0.98.2:47235)
0619 16:52:02.001742 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:02.001786 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:02.002030 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:02.002877 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:02.002878 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:02.003065 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:52:02.003125 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:02.003127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:02.004264 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:02.004265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:02.005124 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:02.005126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:02.005762 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:02.005763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:02.005884 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:02.012698 (+ 6814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:52:02.012707 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:52:02.015242 (+ 2535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:52:02.017913 (+ 2671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:02.017924 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:02.017927 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:02.018270 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:02.019076 (+ 806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:02.019078 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:02.019078 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:02.019247 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:02.019858 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:52:02.019859 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:52:02.019911 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 16:52:02.019966 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 16:52:02.020088 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 16:52:02.020323 15364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c7fca2ca914013983c9aa6e4c13461" candidate_uuid: "6a666372e1ed43b9925eb0d2a61e4891" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0f39a21a9554467bab74fe802df7a1b3" is_pre_election: true
I20260619 16:52:02.020515 15364 raft_consensus.cc:2454] T e0c7fca2ca914013983c9aa6e4c13461 P 0f39a21a9554467bab74fe802df7a1b3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a666372e1ed43b9925eb0d2a61e4891 in term 1.
I20260619 16:52:02.020834 15434 leader_election.cc:302] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f39a21a9554467bab74fe802df7a1b3, 6a666372e1ed43b9925eb0d2a61e4891; no voters:
I20260619 16:52:02.020952 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:02.001832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47235 (local address 127.0.98.3:40877)
0619 16:52:02.002011 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:02.002020 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:02.002058 (+ 38us) server_negotiation.cc:407] Connection header received
0619 16:52:02.002690 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:02.002693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:02.002740 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:02.002871 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:02.003234 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:02.004151 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:02.005221 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:02.005667 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:02.005785 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:02.015422 (+ 9637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:02.015429 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:02.015430 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:02.015448 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:02.017793 (+ 2345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:02.018553 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:02.018562 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:02.018562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:02.018765 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:02.019475 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:02.019477 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:02.019477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:02.019775 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:02.020651 (+ 876us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:02.020723 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 16:52:02.020850 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":71,"threads_started":1}
I20260619 16:52:02.021134 15686 raft_consensus.cc:2781] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 16:52:02.021194 15686 raft_consensus.cc:482] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:52:02.021210 15686 raft_consensus.cc:3037] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 [term 1 FOLLOWER]: Advancing to term 2
W20260619 16:52:02.021831 15433 leader_election.cc:341] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1e4b5295a038445080cc6ec2e3b5ce1d (127.0.98.3:40877): Not found: Tablet not found: e0c7fca2ca914013983c9aa6e4c13461
I20260619 16:52:02.022746 15686 raft_consensus.cc:504] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a666372e1ed43b9925eb0d2a61e4891" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37581 } } peers { permanent_uuid: "1e4b5295a038445080cc6ec2e3b5ce1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40877 } } peers { permanent_uuid: "0f39a21a9554467bab74fe802df7a1b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34561 } }
I20260619 16:52:02.022902 15686 leader_election.cc:288] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 [CANDIDATE]: Term 2 election: Requested vote from peers 1e4b5295a038445080cc6ec2e3b5ce1d (127.0.98.3:40877), 0f39a21a9554467bab74fe802df7a1b3 (127.0.98.1:34561)
I20260619 16:52:02.023088 15364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0c7fca2ca914013983c9aa6e4c13461" candidate_uuid: "6a666372e1ed43b9925eb0d2a61e4891" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0f39a21a9554467bab74fe802df7a1b3"
I20260619 16:52:02.023201 15364 raft_consensus.cc:3037] T e0c7fca2ca914013983c9aa6e4c13461 P 0f39a21a9554467bab74fe802df7a1b3 [term 1 FOLLOWER]: Advancing to term 2
W20260619 16:52:02.023263 15433 leader_election.cc:341] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 1e4b5295a038445080cc6ec2e3b5ce1d (127.0.98.3:40877): Not found: Tablet not found: e0c7fca2ca914013983c9aa6e4c13461
I20260619 16:52:02.024463 15364 raft_consensus.cc:2454] T e0c7fca2ca914013983c9aa6e4c13461 P 0f39a21a9554467bab74fe802df7a1b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a666372e1ed43b9925eb0d2a61e4891 in term 2.
I20260619 16:52:02.024647 15434 leader_election.cc:302] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0f39a21a9554467bab74fe802df7a1b3, 6a666372e1ed43b9925eb0d2a61e4891; no voters: 1e4b5295a038445080cc6ec2e3b5ce1d
I20260619 16:52:02.024745 15686 raft_consensus.cc:2781] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 16:52:02.024870 15686 raft_consensus.cc:686] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 [term 2 LEADER]: Becoming Leader. State: Replica: 6a666372e1ed43b9925eb0d2a61e4891, State: Running, Role: LEADER
I20260619 16:52:02.024966 15686 consensus_queue.cc:237] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a666372e1ed43b9925eb0d2a61e4891" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37581 } } peers { permanent_uuid: "1e4b5295a038445080cc6ec2e3b5ce1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40877 } } peers { permanent_uuid: "0f39a21a9554467bab74fe802df7a1b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34561 } }
I20260619 16:52:02.025266 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:02.025140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.2:59121)
0619 16:52:02.025198 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:02.025233 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 16:52:02.469913 15433 consensus_peers.cc:487] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 -> Peer 1e4b5295a038445080cc6ec2e3b5ce1d (127.0.98.3:40877): Couldn't send request to peer 1e4b5295a038445080cc6ec2e3b5ce1d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e0c7fca2ca914013983c9aa6e4c13461. This is attempt 1: this message will repeat every 5th retry.
I20260619 16:52:02.587949 15364 raft_consensus.cc:1261] T e0c7fca2ca914013983c9aa6e4c13461 P 0f39a21a9554467bab74fe802df7a1b3 [term 2 FOLLOWER]: Refusing update from remote peer 6a666372e1ed43b9925eb0d2a61e4891: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260619 16:52:02.588403 15686 consensus_queue.cc:1035] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f39a21a9554467bab74fe802df7a1b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260619 16:52:02.589466 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:02.589268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.1:55923)
0619 16:52:02.589351 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:02.589410 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62,"thread_start_us":46,"threads_started":1}
I20260619 16:52:02.590638 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:02.590362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.2:49247)
0619 16:52:02.590550 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:02.590589 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":46,"threads_started":1}
I20260619 16:52:02.590660 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:02.590551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.1:52403)
0619 16:52:02.590587 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:02.590625 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260619 16:52:02.815979 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:02.815666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.3:33277)
0619 16:52:02.815863 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:02.815916 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":90,"threads_started":1}
I20260619 16:52:03.018818 15700 ts_tablet_manager.cc:927] T e0c7fca2ca914013983c9aa6e4c13461 P 1e4b5295a038445080cc6ec2e3b5ce1d: Initiating tablet copy from peer 6a666372e1ed43b9925eb0d2a61e4891 (127.0.98.2:37581)
I20260619 16:52:03.019114 15700 tablet_copy_client.cc:316] T e0c7fca2ca914013983c9aa6e4c13461 P 1e4b5295a038445080cc6ec2e3b5ce1d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.2:37581
Jun 19 16:52:03 dist-test-slave-jsw9 krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260619 16:52:03.027978 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:03.019595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60005 (local address 127.0.98.2:37581)
0619 16:52:03.019839 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:03.019844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:03.019855 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:52:03.019885 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:03.019887 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:03.019902 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:03.019996 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:03.020606 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:03.021345 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:03.022158 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:03.022532 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:03.022597 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:03.024986 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:03.024998 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:03.024999 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:03.025007 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:03.026679 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:03.027080 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:03.027084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:03.027084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:03.027145 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:03.027317 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:03.027318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:03.027319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:03.027493 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:03.027571 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:03.027760 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 16:52:03.027857 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":105,"threads_started":1}
I20260619 16:52:03.028002 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:03.019448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37581 (local address 127.0.98.3:60005)
0619 16:52:03.019511 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:03.019534 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:03.019603 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:03.020021 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:03.020030 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:03.020401 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:52:03.020502 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:03.020505 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:03.021431 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:03.021432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:03.022065 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:03.022068 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:03.022603 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:03.022604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:03.022697 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:03.023289 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:52:03.023294 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:52:03.024887 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:52:03.026766 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:03.026777 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:03.026779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:03.027000 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:03.027204 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:03.027205 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:03.027205 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:03.027265 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:03.027553 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:52:03.027561 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:52:03.027629 (+ 68us) client_negotiation.cc:769] Sending connection context
0619 16:52:03.027716 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 16:52:03.027862 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 16:52:03.028568 15519 tablet_copy_service.cc:140] P 6a666372e1ed43b9925eb0d2a61e4891: Received BeginTabletCopySession request for tablet e0c7fca2ca914013983c9aa6e4c13461 from peer 1e4b5295a038445080cc6ec2e3b5ce1d ({username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60005)
I20260619 16:52:03.028646 15519 tablet_copy_service.cc:161] P 6a666372e1ed43b9925eb0d2a61e4891: Beginning new tablet copy session on tablet e0c7fca2ca914013983c9aa6e4c13461 from peer 1e4b5295a038445080cc6ec2e3b5ce1d at {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60005: session id = 1e4b5295a038445080cc6ec2e3b5ce1d-e0c7fca2ca914013983c9aa6e4c13461
I20260619 16:52:03.029193 15519 tablet_copy_source_session.cc:205] T e0c7fca2ca914013983c9aa6e4c13461 P 6a666372e1ed43b9925eb0d2a61e4891: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 16:52:03.030448 15700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0c7fca2ca914013983c9aa6e4c13461. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:52:03.033608 15700 tablet_copy_client.cc:791] T e0c7fca2ca914013983c9aa6e4c13461 P 1e4b5295a038445080cc6ec2e3b5ce1d: tablet copy: Starting download of 0 data blocks...
I20260619 16:52:03.033764 15700 tablet_copy_client.cc:655] T e0c7fca2ca914013983c9aa6e4c13461 P 1e4b5295a038445080cc6ec2e3b5ce1d: tablet copy: Starting download of 1 WAL segments...
I20260619 16:52:03.034627 15700 tablet_copy_client.cc:531] T e0c7fca2ca914013983c9aa6e4c13461 P 1e4b5295a038445080cc6ec2e3b5ce1d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 16:52:03.035538 15700 tablet_bootstrap.cc:492] T e0c7fca2ca914013983c9aa6e4c13461 P 1e4b5295a038445080cc6ec2e3b5ce1d: Bootstrap starting.
I20260619 16:52:03.037617 15700 log.cc:824] T e0c7fca2ca914013983c9aa6e4c13461 P 1e4b5295a038445080cc6ec2e3b5ce1d: Log is configured to *not* fsync() on all Append() calls
I20260619 16:52:03.040593 15700 tablet_bootstrap.cc:492] T e0c7fca2ca914013983c9aa6e4c13461 P 1e4b5295a038445080cc6ec2e3b5ce1d: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:52:03.040854 15700 tablet_bootstrap.cc:492] T e0c7fca2ca914013983c9aa6e4c13461 P 1e4b5295a038445080cc6ec2e3b5ce1d: Bootstrap complete.
I20260619 16:52:03.041004 15700 ts_tablet_manager.cc:1397] T e0c7fca2ca914013983c9aa6e4c13461 P 1e4b5295a038445080cc6ec2e3b5ce1d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260619 16:52:03.042112 15700 raft_consensus.cc:348] T e0c7fca2ca914013983c9aa6e4c13461 P 1e4b5295a038445080cc6ec2e3b5ce1d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a666372e1ed43b9925eb0d2a61e4891" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37581 } } peers { permanent_uuid: "1e4b5295a038445080cc6ec2e3b5ce1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40877 } } peers { permanent_uuid: "0f39a21a9554467bab74fe802df7a1b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34561 } }
I20260619 16:52:03.042337 15700 raft_consensus.cc:729] T e0c7fca2ca914013983c9aa6e4c13461 P 1e4b5295a038445080cc6ec2e3b5ce1d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e4b5295a038445080cc6ec2e3b5ce1d, State: Initialized, Role: FOLLOWER
I20260619 16:52:03.042491 15700 consensus_queue.cc:260] T e0c7fca2ca914013983c9aa6e4c13461 P 1e4b5295a038445080cc6ec2e3b5ce1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a666372e1ed43b9925eb0d2a61e4891" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37581 } } peers { permanent_uuid: "1e4b5295a038445080cc6ec2e3b5ce1d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40877 } } peers { permanent_uuid: "0f39a21a9554467bab74fe802df7a1b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34561 } }
I20260619 16:52:03.042837 15700 ts_tablet_manager.cc:1428] T e0c7fca2ca914013983c9aa6e4c13461 P 1e4b5295a038445080cc6ec2e3b5ce1d: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260619 16:52:03.043243 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:03.043090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.3:42565)
0619 16:52:03.043144 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:03.043202 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 16:52:03.043217 15519 tablet_copy_service.cc:342] P 6a666372e1ed43b9925eb0d2a61e4891: Request end of tablet copy session 1e4b5295a038445080cc6ec2e3b5ce1d-e0c7fca2ca914013983c9aa6e4c13461 received from {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60005
I20260619 16:52:03.043293 15519 tablet_copy_service.cc:433] P 6a666372e1ed43b9925eb0d2a61e4891: ending tablet copy session 1e4b5295a038445080cc6ec2e3b5ce1d-e0c7fca2ca914013983c9aa6e4c13461 on tablet e0c7fca2ca914013983c9aa6e4c13461 with peer 1e4b5295a038445080cc6ec2e3b5ce1d
I20260619 16:52:03.043637 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:03.043504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.3:58397)
0619 16:52:03.043575 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:03.043610 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 16:52:03.582046 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:03.581794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.2:58327)
0619 16:52:03.581937 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:03.581972 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":57,"threads_started":1}
I20260619 16:52:03.582038 15633 raft_consensus.cc:1203] T e0c7fca2ca914013983c9aa6e4c13461 P 1e4b5295a038445080cc6ec2e3b5ce1d [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260619 16:52:03.582697 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:03.582505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.3:40431)
0619 16:52:03.582614 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:03.582640 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":51,"threads_started":1}
I20260619 16:52:03.591462 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:03.591197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.1:45171)
0619 16:52:03.591350 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:03.591403 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":76,"threads_started":1}
I20260619 16:52:04.583102 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:04.582659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.2:38115)
0619 16:52:04.582960 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:04.583022 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":96,"threads_started":1}
I20260619 16:52:04.583549 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:04.583240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.3:39135)
0619 16:52:04.583444 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:04.583491 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":96,"threads_started":1}
I20260619 16:52:04.592315 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:04.592048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.1:42553)
0619 16:52:04.592213 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:04.592254 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":81,"threads_started":1}
I20260619 16:52:05.584373 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:05.584088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.3:44023)
0619 16:52:05.584267 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:05.584314 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":84,"threads_started":1}
I20260619 16:52:05.584373 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:05.584072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.2:51919)
0619 16:52:05.584267 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:05.584313 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":98,"threads_started":1}
I20260619 16:52:05.593312 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:05.593009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.1:50223)
0619 16:52:05.593186 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:05.593244 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":92,"threads_started":1}
I20260619 16:52:06.585286 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:06.584986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.3:44627)
0619 16:52:06.585159 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:06.585206 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
I20260619 16:52:06.585547 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:06.584983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.2:43967)
0619 16:52:06.585442 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:06.585481 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":90,"threads_started":1}
I20260619 16:52:06.586722 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:06.586595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.2:55439)
0619 16:52:06.586659 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:06.586687 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 16:52:06.594264 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:06.593955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.1:48809)
0619 16:52:06.594119 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:06.594171 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":81,"threads_started":1}
I20260619 16:52:07.586337 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:07.585993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.3:47715)
0619 16:52:07.586152 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:07.586256 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":77,"threads_started":1}
I20260619 16:52:07.587348 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:07.587096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.2:60007)
0619 16:52:07.587240 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:07.587281 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":64,"threads_started":1}
I20260619 16:52:07.595074 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:07.594810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.1:34599)
0619 16:52:07.594971 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:07.595015 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":77,"threads_started":1}
I20260619 16:52:08.587173 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:08.586897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.3:55677)
0619 16:52:08.587061 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:08.587106 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":87,"threads_started":1}
I20260619 16:52:08.588063 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:08.587825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.2:60477)
0619 16:52:08.587976 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:08.588016 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":71,"threads_started":1}
I20260619 16:52:08.596199 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:08.595897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.1:49873)
0619 16:52:08.596081 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:08.596135 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":83,"threads_started":1}
I20260619 16:52:09.588056 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:09.587765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.3:40743)
0619 16:52:09.587936 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:09.587989 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":83,"threads_started":1}
I20260619 16:52:09.588755 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:09.588493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.2:48071)
0619 16:52:09.588673 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:09.588712 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":84,"threads_started":1}
I20260619 16:52:09.597163 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:09.596883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.1:33543)
0619 16:52:09.597052 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:09.597104 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":80,"threads_started":1}
I20260619 16:52:10.588863 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:10.588580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.3:60491)
0619 16:52:10.588739 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:10.588780 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":99,"threads_started":1}
I20260619 16:52:10.589567 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:10.589291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.2:35735)
0619 16:52:10.589474 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:10.589511 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":67,"threads_started":1}
I20260619 16:52:10.599472 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:10.599053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.1:58583)
0619 16:52:10.599191 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:10.599235 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
I20260619 16:52:11.590237 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:11.589851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.3:35157)
0619 16:52:11.590061 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:11.590128 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":112,"threads_started":1}
I20260619 16:52:11.590519 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:11.590312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.2:58765)
0619 16:52:11.590445 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:11.590479 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":66,"threads_started":1}
I20260619 16:52:11.600513 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:11.600209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.1:35711)
0619 16:52:11.600391 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:11.600453 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":99,"threads_started":1}
I20260619 16:52:12.591369 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:12.591044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.3:57101)
0619 16:52:12.591256 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:12.591305 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":116,"threads_started":1}
I20260619 16:52:12.591385 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:12.591009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.2:46023)
0619 16:52:12.591207 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:12.591261 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":84,"threads_started":1}
I20260619 16:52:12.601521 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:12.601193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.1:53809)
0619 16:52:12.601384 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:12.601452 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":85,"threads_started":1}
I20260619 16:52:13.592232 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:13.591943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.2:60087)
0619 16:52:13.592131 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:13.592180 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":92,"threads_started":1}
I20260619 16:52:13.592232 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:13.591943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.3:48967)
0619 16:52:13.592127 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:13.592166 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":90,"threads_started":1}
I20260619 16:52:13.602366 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:13.602041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.1:33189)
0619 16:52:13.602259 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:13.602306 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":95,"threads_started":1}
I20260619 16:52:14.593149 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:14.592846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.2:51449)
0619 16:52:14.593033 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:14.593077 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":95,"threads_started":1}
I20260619 16:52:14.593142 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:14.592844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.3:44477)
0619 16:52:14.593029 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:14.593077 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":90,"threads_started":1}
I20260619 16:52:14.603235 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:14.602936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.1:42861)
0619 16:52:14.603109 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:14.603161 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":83,"threads_started":1}
I20260619 16:52:15.593889 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:15.593609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.3:39993)
0619 16:52:15.593769 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:15.593809 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":130,"threads_started":1}
I20260619 16:52:15.594035 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:15.593826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.2:53425)
0619 16:52:15.593943 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:15.593979 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
I20260619 16:52:15.604259 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:15.603971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37483 (local address 127.0.98.1:51549)
0619 16:52:15.604165 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:15.604200 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":100,"threads_started":1}
I20260619 16:52:15.817276 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 15251
I20260619 16:52:15.822454 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 15414
I20260619 16:52:15.827389 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 15549
2026-06-19T16:52:15Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18562 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:52:15 dist-test-slave-jsw9 krb5kdc[15752](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:52:15 dist-test-slave-jsw9 krb5kdc[15752](info): set up 2 sockets
Jun 19 16:52:15 dist-test-slave-jsw9 krb5kdc[15752](info): commencing operation
krb5kdc: starting...
W20260619 16:52:16.818491 392 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:52:16 dist-test-slave-jsw9 krb5kdc[15752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:52:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:52:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:52:16.949792 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41581
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:52:17.041029 15765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:17.041307 15765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:17.041343 15765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:17.042688 15765 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 16:52:17.042735 15765 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 16:52:17.042752 15765 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 16:52:17.042768 15765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:52:17.042784 15765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:17.042801 15765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:17.042829 15765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:52:17.042851 15765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:52:17.044514 15765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:52:17.044968 15765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:17.045235 15765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:52:17.050823 15765 server_base.cc:1034] running on GCE node
W20260619 16:52:17.050947 15773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:17.050947 15771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:17.051085 15770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:17.051455 15765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:17.051781 15765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:17.052961 15765 hybrid_clock.cc:648] HybridClock initialized: now 1781887937052941 us; error 36 us; skew 500 ppm
Jun 19 16:52:17 dist-test-slave-jsw9 krb5kdc[15752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:17.055966 15765 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:52:17.056442 15765 webserver.cc:460] Webserver started at http://127.0.98.62:35829/ using document root <none> and password file <none>
I20260619 16:52:17.056681 15765 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:17.056733 15765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:17.056869 15765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:52:17.058055 15765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "d81eda6159264f73977473029f411450"
format_stamp: "Formatted at 2026-06-19 16:52:17 on dist-test-slave-jsw9"
server_key: "89fccf1af9c4e617c7e3b2083adcd4c2"
server_key_iv: "1307d26a6a5e9351f5ffa16780311094"
server_key_version: "encryptionkey@0"
I20260619 16:52:17.058461 15765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "d81eda6159264f73977473029f411450"
format_stamp: "Formatted at 2026-06-19 16:52:17 on dist-test-slave-jsw9"
server_key: "89fccf1af9c4e617c7e3b2083adcd4c2"
server_key_iv: "1307d26a6a5e9351f5ffa16780311094"
server_key_version: "encryptionkey@0"
I20260619 16:52:17.060343 15765 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:52:17.061295 15780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:17.061740 15765 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:52:17.061848 15765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "d81eda6159264f73977473029f411450"
format_stamp: "Formatted at 2026-06-19 16:52:17 on dist-test-slave-jsw9"
server_key: "89fccf1af9c4e617c7e3b2083adcd4c2"
server_key_iv: "1307d26a6a5e9351f5ffa16780311094"
server_key_version: "encryptionkey@0"
I20260619 16:52:17.061928 15765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:17.074232 15765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:17.077194 15765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:17.077616 15765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:17.083345 15765 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41581
I20260619 16:52:17.083420 15832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41581 every 8 connection(s)
I20260619 16:52:17.083911 15765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:52:17.084906 15833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:52:17.085932 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 15765
I20260619 16:52:17.086042 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:52:17.086293 392 external_mini_cluster.cc:1428] Setting key a3d6e530d3eecc3dedc9982210f6fee8
I20260619 16:52:17.088770 15833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450: Bootstrap starting.
I20260619 16:52:17.089488 15833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450: Neither blocks nor log segments found. Creating new log.
I20260619 16:52:17.089828 15833 log.cc:824] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450: Log is configured to *not* fsync() on all Append() calls
I20260619 16:52:17.090705 15833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450: No bootstrap required, opened a new log
I20260619 16:52:17.092903 15833 raft_consensus.cc:348] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d81eda6159264f73977473029f411450" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41581 } }
I20260619 16:52:17.093078 15833 raft_consensus.cc:374] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:52:17.093104 15833 raft_consensus.cc:729] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d81eda6159264f73977473029f411450, State: Initialized, Role: FOLLOWER
I20260619 16:52:17.093236 15833 consensus_queue.cc:260] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d81eda6159264f73977473029f411450" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41581 } }
I20260619 16:52:17.093370 15833 raft_consensus.cc:388] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:52:17.093416 15833 raft_consensus.cc:482] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:52:17.093462 15833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 16:52:17 dist-test-slave-jsw9 krb5kdc[15752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:52:17.094240 15833 raft_consensus.cc:504] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d81eda6159264f73977473029f411450" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41581 } }
I20260619 16:52:17.094357 15833 leader_election.cc:302] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d81eda6159264f73977473029f411450; no voters:
I20260619 16:52:17.094565 15833 leader_election.cc:288] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:52:17.094606 15838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:52:17.094732 15838 raft_consensus.cc:686] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [term 1 LEADER]: Becoming Leader. State: Replica: d81eda6159264f73977473029f411450, State: Running, Role: LEADER
I20260619 16:52:17.094828 15838 consensus_queue.cc:237] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d81eda6159264f73977473029f411450" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41581 } }
I20260619 16:52:17.094926 15833 sys_catalog.cc:564] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:52:17.095227 15838 sys_catalog.cc:455] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d81eda6159264f73977473029f411450. Latest consensus state: current_term: 1 leader_uuid: "d81eda6159264f73977473029f411450" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d81eda6159264f73977473029f411450" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41581 } } }
I20260619 16:52:17.095378 15838 sys_catalog.cc:458] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [sys.catalog]: This master's current role is: LEADER
I20260619 16:52:17.095522 15839 sys_catalog.cc:455] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d81eda6159264f73977473029f411450" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d81eda6159264f73977473029f411450" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41581 } } }
I20260619 16:52:17.095666 15844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:52:17.096352 15844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:52:17.095680 15839 sys_catalog.cc:458] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [sys.catalog]: This master's current role is: LEADER
I20260619 16:52:17.098173 15844 catalog_manager.cc:1347] Generated new cluster ID: 13220e2f0c844b62beb7fd4c6d13a2ad
I20260619 16:52:17.098256 15844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:52:17.098832 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.086997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49998 (local address 127.0.98.62:41581)
0619 16:52:17.087174 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:17.087183 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:17.087212 (+ 29us) server_negotiation.cc:407] Connection header received
0619 16:52:17.087701 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:17.087709 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:17.087763 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:17.087891 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:17.088778 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.089868 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.090991 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.091479 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.091596 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:17.094419 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:17.094424 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:17.094425 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:17.094441 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:17.097052 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:17.097508 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:17.097511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:17.097512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:17.097611 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:17.097840 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:17.097841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:17.097841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:17.098113 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:17.098322 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:17.098397 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 16:52:17.098660 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":75,"threads_started":1}
I20260619 16:52:17.111202 15844 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:52:17.111860 15844 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:52:17.117774 15844 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450: Generated new TSK 0
I20260619 16:52:17.118021 15844 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:52:17.150312 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41581
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:52:17.254632 15859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:17.254941 15859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:17.254980 15859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:17.256465 15859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:17.256528 15859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:17.256573 15859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:52:17.258467 15859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:52:17.258793 15859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:17.259061 15859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:17.265187 15867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:17.265286 15859 server_base.cc:1034] running on GCE node
W20260619 16:52:17.265169 15865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:17.265165 15864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:17.265758 15859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:17.266108 15859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:17.267326 15859 hybrid_clock.cc:648] HybridClock initialized: now 1781887937267313 us; error 50 us; skew 500 ppm
Jun 19 16:52:17 dist-test-slave-jsw9 krb5kdc[15752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:17.270628 15859 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:52:17.271190 15859 webserver.cc:460] Webserver started at http://127.0.98.1:36105/ using document root <none> and password file <none>
I20260619 16:52:17.271440 15859 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:17.271494 15859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:17.271644 15859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:52:17.272964 15859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/data/instance:
uuid: "6f7a800440a14ac9a5a0cf1baa974e02"
format_stamp: "Formatted at 2026-06-19 16:52:17 on dist-test-slave-jsw9"
server_key: "26d313b6c765283e576cb1d772b2df45"
server_key_iv: "2e64450e2059689a096657a292862996"
server_key_version: "encryptionkey@0"
I20260619 16:52:17.273488 15859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6f7a800440a14ac9a5a0cf1baa974e02"
format_stamp: "Formatted at 2026-06-19 16:52:17 on dist-test-slave-jsw9"
server_key: "26d313b6c765283e576cb1d772b2df45"
server_key_iv: "2e64450e2059689a096657a292862996"
server_key_version: "encryptionkey@0"
I20260619 16:52:17.275170 15859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260619 16:52:17.276185 15874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:17.276468 15859 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:52:17.276571 15859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/wal
uuid: "6f7a800440a14ac9a5a0cf1baa974e02"
format_stamp: "Formatted at 2026-06-19 16:52:17 on dist-test-slave-jsw9"
server_key: "26d313b6c765283e576cb1d772b2df45"
server_key_iv: "2e64450e2059689a096657a292862996"
server_key_version: "encryptionkey@0"
I20260619 16:52:17.276676 15859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:17.297698 15859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:17.300266 15859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:17.300416 15859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:17.300645 15859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:52:17.301013 15859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:52:17.301049 15859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:17.301082 15859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:52:17.301100 15859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:17.308264 15859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34985
I20260619 16:52:17.308370 15987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34985 every 8 connection(s)
I20260619 16:52:17.308719 15859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 16:52:17 dist-test-slave-jsw9 krb5kdc[15752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:52:17.317948 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 15859
I20260619 16:52:17.318076 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:52:17.318332 392 external_mini_cluster.cc:1428] Setting key 0cf9399ced4f02147d469bfd5898f56f
I20260619 16:52:17.319886 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.309856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38925 (local address 127.0.98.62:41581)
0619 16:52:17.309921 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:17.309923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:17.310148 (+ 225us) server_negotiation.cc:407] Connection header received
0619 16:52:17.310833 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:17.310835 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:17.310849 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:17.310904 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:17.312025 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.312578 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.313669 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.313900 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.313960 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:17.316617 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:17.316619 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:17.316620 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:17.316628 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:17.318103 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:17.318682 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:17.318684 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:17.318684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:17.318749 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:17.319000 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:17.319001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:17.319001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:17.319156 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:17.319236 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:17.319705 (+ 469us) server_negotiation.cc:299] Negotiation successful
0619 16:52:17.319808 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 16:52:17.320185 15990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.309779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.1:38925)
0619 16:52:17.310008 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:17.310055 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:17.310713 (+ 658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:17.310990 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:17.310996 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:17.311670 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:52:17.311914 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:17.311920 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.312675 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.312676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:17.313574 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:17.313577 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.313979 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.313979 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:17.314128 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:17.314778 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:52:17.314789 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:52:17.316480 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:52:17.318240 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:17.318252 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:17.318255 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:17.318604 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:17.318815 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:17.318816 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:17.318816 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:17.318937 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:17.319256 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:52:17.319276 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:52:17.319497 (+ 221us) client_negotiation.cc:769] Sending connection context
0619 16:52:17.319629 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 16:52:17.319883 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":125,"threads_started":1}
I20260619 16:52:17.320969 15988 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41581
I20260619 16:52:17.321197 15988 heartbeater.cc:464] Registering TS with master...
I20260619 16:52:17.321614 15988 heartbeater.cc:511] Master 127.0.98.62:41581 requested a full tablet report, sending...
I20260619 16:52:17.322373 15797 ts_manager.cc:194] Registered new tserver with Master: 6f7a800440a14ac9a5a0cf1baa974e02 (127.0.98.1:34985)
I20260619 16:52:17.323202 15797 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38925
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:52:17.350636 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41581
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:52:17.445370 15993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:17.445645 15993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:17.445673 15993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:17.447031 15993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:17.447088 15993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:17.447118 15993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:52:17.448676 15993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:52:17.448925 15993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:17.449143 15993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:17.454860 16001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:17.454912 15993 server_base.cc:1034] running on GCE node
W20260619 16:52:17.454890 15998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:17.454855 15999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:17.455302 15993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:17.455576 15993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:17.456751 15993 hybrid_clock.cc:648] HybridClock initialized: now 1781887937456739 us; error 31 us; skew 500 ppm
Jun 19 16:52:17 dist-test-slave-jsw9 krb5kdc[15752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:17.459497 15993 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:52:17.459986 15993 webserver.cc:460] Webserver started at http://127.0.98.2:44573/ using document root <none> and password file <none>
I20260619 16:52:17.460186 15993 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:17.460228 15993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:17.460335 15993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:52:17.461539 15993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/data/instance:
uuid: "46b3ce200da840748de5cffca4a76fd0"
format_stamp: "Formatted at 2026-06-19 16:52:17 on dist-test-slave-jsw9"
server_key: "a7c927e12c01e5230609af3ae41aefb3"
server_key_iv: "5c63f9b0c2237301d74a5d2183a026d5"
server_key_version: "encryptionkey@0"
I20260619 16:52:17.461871 15993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/wal/instance:
uuid: "46b3ce200da840748de5cffca4a76fd0"
format_stamp: "Formatted at 2026-06-19 16:52:17 on dist-test-slave-jsw9"
server_key: "a7c927e12c01e5230609af3ae41aefb3"
server_key_iv: "5c63f9b0c2237301d74a5d2183a026d5"
server_key_version: "encryptionkey@0"
I20260619 16:52:17.463231 15993 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 16:52:17.463999 16008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:17.464183 15993 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:52:17.464272 15993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/wal
uuid: "46b3ce200da840748de5cffca4a76fd0"
format_stamp: "Formatted at 2026-06-19 16:52:17 on dist-test-slave-jsw9"
server_key: "a7c927e12c01e5230609af3ae41aefb3"
server_key_iv: "5c63f9b0c2237301d74a5d2183a026d5"
server_key_version: "encryptionkey@0"
I20260619 16:52:17.464383 15993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:17.495388 15993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:17.497743 15993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:17.497874 15993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:17.498113 15993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:52:17.498481 15993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:52:17.498520 15993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:17.498553 15993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:52:17.498571 15993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:17.505558 15993 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45561
I20260619 16:52:17.505646 16121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45561 every 8 connection(s)
I20260619 16:52:17.506004 15993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:52:17.506843 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 15993
I20260619 16:52:17.506955 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:52:17.507174 392 external_mini_cluster.cc:1428] Setting key 8de30dcb062bcf092c238510ce30c599
Jun 19 16:52:17 dist-test-slave-jsw9 krb5kdc[15752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 16:52:17.516994 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.506964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55577 (local address 127.0.98.62:41581)
0619 16:52:17.507024 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:17.507027 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:17.507544 (+ 517us) server_negotiation.cc:407] Connection header received
0619 16:52:17.508124 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:17.508125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:17.508138 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:17.508192 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:17.509249 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.509821 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.510965 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.511226 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.511293 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:17.513899 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:17.513901 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:17.513901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:17.513909 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:17.515300 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:17.515835 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:17.515836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:17.515836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:17.515892 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:17.516209 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:17.516210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:17.516210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:17.516377 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:17.516472 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:17.516782 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 16:52:17.516883 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 16:52:17.517253 16124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.506944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.2:55577)
0619 16:52:17.507406 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:17.507450 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:17.508021 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:17.508278 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:17.508284 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:17.508906 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:52:17.509150 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:17.509156 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.509935 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.509936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:17.510881 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:17.510885 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.511314 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.511315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:17.511452 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:17.512058 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:52:17.512068 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:52:17.513785 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:52:17.515390 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:17.515398 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:17.515401 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:17.515740 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:17.515958 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:17.515959 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:17.515959 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:17.516110 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:17.516470 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:52:17.516480 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:52:17.516643 (+ 163us) client_negotiation.cc:769] Sending connection context
0619 16:52:17.516758 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 16:52:17.516952 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":90,"threads_started":1}
I20260619 16:52:17.517823 16122 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41581
I20260619 16:52:17.517935 16122 heartbeater.cc:464] Registering TS with master...
I20260619 16:52:17.518169 16122 heartbeater.cc:511] Master 127.0.98.62:41581 requested a full tablet report, sending...
I20260619 16:52:17.518642 15797 ts_manager.cc:194] Registered new tserver with Master: 46b3ce200da840748de5cffca4a76fd0 (127.0.98.2:45561)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 16:52:17.519104 15797 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55577
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:52:17.539637 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41581
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:52:17.634332 16127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:17.634614 16127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:17.634644 16127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:17.636037 16127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:17.636101 16127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:17.636133 16127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:52:17.637998 16127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:52:17.638329 16127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:17.638607 16127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:17.644011 16132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:17.644011 16133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:17.644014 16135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:17.644130 16127 server_base.cc:1034] running on GCE node
I20260619 16:52:17.644523 16127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:17.644786 16127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:17.645956 16127 hybrid_clock.cc:648] HybridClock initialized: now 1781887937645939 us; error 35 us; skew 500 ppm
Jun 19 16:52:17 dist-test-slave-jsw9 krb5kdc[15752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:17.648893 16127 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:52:17.649406 16127 webserver.cc:460] Webserver started at http://127.0.98.3:36917/ using document root <none> and password file <none>
I20260619 16:52:17.649662 16127 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:17.649716 16127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:17.649852 16127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:52:17.651149 16127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/data/instance:
uuid: "40dabdafed9140fe9787acd08ba6e484"
format_stamp: "Formatted at 2026-06-19 16:52:17 on dist-test-slave-jsw9"
server_key: "f025c98b58d1b72e43696426d199f462"
server_key_iv: "b963d1252c920a56cbc7bfebc7768b4e"
server_key_version: "encryptionkey@0"
I20260619 16:52:17.651512 16127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/wal/instance:
uuid: "40dabdafed9140fe9787acd08ba6e484"
format_stamp: "Formatted at 2026-06-19 16:52:17 on dist-test-slave-jsw9"
server_key: "f025c98b58d1b72e43696426d199f462"
server_key_iv: "b963d1252c920a56cbc7bfebc7768b4e"
server_key_version: "encryptionkey@0"
I20260619 16:52:17.652972 16127 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 16:52:17.653851 16142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:17.654014 16127 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:52:17.654109 16127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/wal
uuid: "40dabdafed9140fe9787acd08ba6e484"
format_stamp: "Formatted at 2026-06-19 16:52:17 on dist-test-slave-jsw9"
server_key: "f025c98b58d1b72e43696426d199f462"
server_key_iv: "b963d1252c920a56cbc7bfebc7768b4e"
server_key_version: "encryptionkey@0"
I20260619 16:52:17.654179 16127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:17.683818 16127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:17.686439 16127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:17.686626 16127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:17.686919 16127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:52:17.687289 16127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:52:17.687327 16127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:17.687367 16127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:52:17.687397 16127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:17.694736 16127 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39447
I20260619 16:52:17.694804 16255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39447 every 8 connection(s)
I20260619 16:52:17.695186 16127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:52:17.696187 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 16127
I20260619 16:52:17.696303 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:52:17.696530 392 external_mini_cluster.cc:1428] Setting key da0fe3a172fb9d0469434e0cfbb3de48
Jun 19 16:52:17 dist-test-slave-jsw9 krb5kdc[15752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:52:17.705511 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.696102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46143 (local address 127.0.98.62:41581)
0619 16:52:17.696173 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:17.696175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:17.696393 (+ 218us) server_negotiation.cc:407] Connection header received
0619 16:52:17.697185 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:17.697186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:17.697201 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:17.697269 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:17.698481 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.699017 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.700156 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.700361 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.700440 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:17.702873 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:17.702875 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:17.702875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:17.702883 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:17.704018 (+ 1135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:17.704452 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:17.704453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:17.704453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:17.704525 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:17.704796 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:17.704797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:17.704797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:17.704947 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:17.705013 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:17.705347 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 16:52:17.705432 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 16:52:17.705727 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.696042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.3:46143)
0619 16:52:17.696232 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:17.696288 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:17.697065 (+ 777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:17.697352 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:17.697357 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:17.698062 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:52:17.698357 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:17.698363 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.699117 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.699118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:17.700045 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:17.700048 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.700468 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.700469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:17.700609 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:17.701248 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:52:17.701257 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:52:17.702786 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:52:17.704104 (+ 1318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:17.704111 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:17.704114 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:17.704344 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:17.704605 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:17.704605 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:17.704605 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:17.704708 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:17.705042 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:52:17.705050 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:52:17.705213 (+ 163us) client_negotiation.cc:769] Sending connection context
0619 16:52:17.705318 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 16:52:17.705467 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":102,"threads_started":1}
I20260619 16:52:17.706319 16256 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41581
I20260619 16:52:17.706431 16256 heartbeater.cc:464] Registering TS with master...
I20260619 16:52:17.706607 16256 heartbeater.cc:511] Master 127.0.98.62:41581 requested a full tablet report, sending...
I20260619 16:52:17.707036 15797 ts_manager.cc:194] Registered new tserver with Master: 40dabdafed9140fe9787acd08ba6e484 (127.0.98.3:39447)
I20260619 16:52:17.707415 15797 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46143
I20260619 16:52:17.708331 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:52:17.716496 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.709788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50006 (local address 127.0.98.62:41581)
0619 16:52:17.709836 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:17.709838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:17.709938 (+ 100us) server_negotiation.cc:407] Connection header received
0619 16:52:17.709975 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:17.709976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:17.709986 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:17.710047 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:17.710640 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.711147 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.711931 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.712132 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.712196 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:17.713428 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:17.713429 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:17.713430 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:17.713437 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:17.715183 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:17.715559 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:17.715560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:17.715560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:17.715629 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:17.715897 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:17.715899 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:17.715899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:17.716097 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:17.716193 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:17.716336 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 16:52:17.716410 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 16:52:17.718390 15797 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:52:17.720418 15797 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:52:17.727448 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.723265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53830 (local address 127.0.98.1:34985)
0619 16:52:17.723743 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:17.723747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:17.723755 (+ 8us) server_negotiation.cc:407] Connection header received
0619 16:52:17.723787 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:17.723789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:17.723800 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:17.724053 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:52:17.724134 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.725177 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.726464 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.726997 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.727075 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:17.727147 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:17.727288 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 16:52:17.727340 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":90,"threads_started":1}
I20260619 16:52:17.727434 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.723155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34985 (local address 127.0.0.1:53830)
0619 16:52:17.723616 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:17.723624 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:17.723659 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:17.723886 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:17.723887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:17.723889 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:52:17.723981 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:17.723983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.725281 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.725282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:17.726392 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:17.726394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.727079 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.727080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:17.727177 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:17.727193 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 16:52:17.727271 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 16:52:17.727339 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":4,"thread_start_us":103,"threads_started":1}
I20260619 16:52:17.727591 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.723145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45561 (local address 127.0.0.1:54068)
0619 16:52:17.723347 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:17.723397 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:17.723478 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:17.723826 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:17.723830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:17.723833 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:52:17.723929 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:17.723932 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.725018 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.725019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:17.726379 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:17.726382 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.727365 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.727367 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:17.727431 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:17.727435 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:52:17.727463 (+ 28us) client_negotiation.cc:240] Negotiation successful
0619 16:52:17.727509 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":77,"threads_started":1}
I20260619 16:52:17.727782 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.723436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49054 (local address 127.0.98.3:39447)
0619 16:52:17.724004 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:17.724009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:17.724020 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:52:17.724063 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:17.724065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:17.724080 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:17.724154 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:52:17.724403 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.725434 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.726516 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.727267 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.727338 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:17.727396 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:17.727632 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 16:52:17.727687 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":97,"threads_started":1}
I20260619 16:52:17.727953 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.723447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54068 (local address 127.0.98.2:45561)
0619 16:52:17.723654 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:17.723659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:17.723669 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:52:17.723710 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:17.723712 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:17.723726 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:17.723809 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:52:17.724018 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.724919 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.726505 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.727268 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.727728 (+ 460us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:17.727805 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:17.727833 (+ 28us) server_negotiation.cc:299] Negotiation successful
0619 16:52:17.727882 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":84,"threads_started":1}
I20260619 16:52:17.728085 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.723155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39447 (local address 127.0.0.1:49054)
0619 16:52:17.723443 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:17.723452 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:17.723498 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:17.724263 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:17.724264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:17.724266 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:52:17.724315 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:17.724317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.725526 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.725536 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:17.726445 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:17.726448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.727426 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.727427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:17.727480 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:17.727482 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 16:52:17.727521 (+ 39us) client_negotiation.cc:240] Negotiation successful
0619 16:52:17.727555 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":100,"threads_started":1}
I20260619 16:52:17.728811 16056 tablet_service.cc:1467] Processing CreateTablet for tablet 084ea478d8b1456a9e071c1c2916bb7d (DEFAULT_TABLE table=test-table [id=ec3f9ae58d294e9e8fe6edce21c6012a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:52:17.728897 16190 tablet_service.cc:1467] Processing CreateTablet for tablet 084ea478d8b1456a9e071c1c2916bb7d (DEFAULT_TABLE table=test-table [id=ec3f9ae58d294e9e8fe6edce21c6012a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:52:17.728899 15922 tablet_service.cc:1467] Processing CreateTablet for tablet 084ea478d8b1456a9e071c1c2916bb7d (DEFAULT_TABLE table=test-table [id=ec3f9ae58d294e9e8fe6edce21c6012a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:52:17.729218 16056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 084ea478d8b1456a9e071c1c2916bb7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:52:17.729229 16190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 084ea478d8b1456a9e071c1c2916bb7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:52:17.729229 15922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 084ea478d8b1456a9e071c1c2916bb7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:52:17.732353 16273 tablet_bootstrap.cc:492] T 084ea478d8b1456a9e071c1c2916bb7d P 46b3ce200da840748de5cffca4a76fd0: Bootstrap starting.
I20260619 16:52:17.732746 16275 tablet_bootstrap.cc:492] T 084ea478d8b1456a9e071c1c2916bb7d P 40dabdafed9140fe9787acd08ba6e484: Bootstrap starting.
I20260619 16:52:17.732999 16274 tablet_bootstrap.cc:492] T 084ea478d8b1456a9e071c1c2916bb7d P 6f7a800440a14ac9a5a0cf1baa974e02: Bootstrap starting.
I20260619 16:52:17.733484 16275 tablet_bootstrap.cc:654] T 084ea478d8b1456a9e071c1c2916bb7d P 40dabdafed9140fe9787acd08ba6e484: Neither blocks nor log segments found. Creating new log.
I20260619 16:52:17.733484 16273 tablet_bootstrap.cc:654] T 084ea478d8b1456a9e071c1c2916bb7d P 46b3ce200da840748de5cffca4a76fd0: Neither blocks nor log segments found. Creating new log.
I20260619 16:52:17.733579 16274 tablet_bootstrap.cc:654] T 084ea478d8b1456a9e071c1c2916bb7d P 6f7a800440a14ac9a5a0cf1baa974e02: Neither blocks nor log segments found. Creating new log.
I20260619 16:52:17.734117 16274 log.cc:824] T 084ea478d8b1456a9e071c1c2916bb7d P 6f7a800440a14ac9a5a0cf1baa974e02: Log is configured to *not* fsync() on all Append() calls
I20260619 16:52:17.734164 16275 log.cc:824] T 084ea478d8b1456a9e071c1c2916bb7d P 40dabdafed9140fe9787acd08ba6e484: Log is configured to *not* fsync() on all Append() calls
I20260619 16:52:17.734164 16273 log.cc:824] T 084ea478d8b1456a9e071c1c2916bb7d P 46b3ce200da840748de5cffca4a76fd0: Log is configured to *not* fsync() on all Append() calls
I20260619 16:52:17.734926 16274 tablet_bootstrap.cc:492] T 084ea478d8b1456a9e071c1c2916bb7d P 6f7a800440a14ac9a5a0cf1baa974e02: No bootstrap required, opened a new log
I20260619 16:52:17.734999 16274 ts_tablet_manager.cc:1397] T 084ea478d8b1456a9e071c1c2916bb7d P 6f7a800440a14ac9a5a0cf1baa974e02: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:52:17.735054 16273 tablet_bootstrap.cc:492] T 084ea478d8b1456a9e071c1c2916bb7d P 46b3ce200da840748de5cffca4a76fd0: No bootstrap required, opened a new log
I20260619 16:52:17.735095 16275 tablet_bootstrap.cc:492] T 084ea478d8b1456a9e071c1c2916bb7d P 40dabdafed9140fe9787acd08ba6e484: No bootstrap required, opened a new log
I20260619 16:52:17.735126 16273 ts_tablet_manager.cc:1397] T 084ea478d8b1456a9e071c1c2916bb7d P 46b3ce200da840748de5cffca4a76fd0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:52:17.735150 16275 ts_tablet_manager.cc:1397] T 084ea478d8b1456a9e071c1c2916bb7d P 40dabdafed9140fe9787acd08ba6e484: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:52:17.736541 16274 raft_consensus.cc:348] T 084ea478d8b1456a9e071c1c2916bb7d P 6f7a800440a14ac9a5a0cf1baa974e02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40dabdafed9140fe9787acd08ba6e484" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39447 } } peers { permanent_uuid: "46b3ce200da840748de5cffca4a76fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "6f7a800440a14ac9a5a0cf1baa974e02" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34985 } }
I20260619 16:52:17.736727 16274 raft_consensus.cc:374] T 084ea478d8b1456a9e071c1c2916bb7d P 6f7a800440a14ac9a5a0cf1baa974e02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:52:17.736761 16274 raft_consensus.cc:729] T 084ea478d8b1456a9e071c1c2916bb7d P 6f7a800440a14ac9a5a0cf1baa974e02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f7a800440a14ac9a5a0cf1baa974e02, State: Initialized, Role: FOLLOWER
I20260619 16:52:17.736893 16274 consensus_queue.cc:260] T 084ea478d8b1456a9e071c1c2916bb7d P 6f7a800440a14ac9a5a0cf1baa974e02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40dabdafed9140fe9787acd08ba6e484" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39447 } } peers { permanent_uuid: "46b3ce200da840748de5cffca4a76fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "6f7a800440a14ac9a5a0cf1baa974e02" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34985 } }
I20260619 16:52:17.737234 16274 ts_tablet_manager.cc:1428] T 084ea478d8b1456a9e071c1c2916bb7d P 6f7a800440a14ac9a5a0cf1baa974e02: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:52:17.737239 15988 heartbeater.cc:503] Master 127.0.98.62:41581 was elected leader, sending a full tablet report...
I20260619 16:52:17.737249 16275 raft_consensus.cc:348] T 084ea478d8b1456a9e071c1c2916bb7d P 40dabdafed9140fe9787acd08ba6e484 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40dabdafed9140fe9787acd08ba6e484" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39447 } } peers { permanent_uuid: "46b3ce200da840748de5cffca4a76fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "6f7a800440a14ac9a5a0cf1baa974e02" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34985 } }
I20260619 16:52:17.737437 16275 raft_consensus.cc:374] T 084ea478d8b1456a9e071c1c2916bb7d P 40dabdafed9140fe9787acd08ba6e484 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:52:17.737514 16275 raft_consensus.cc:729] T 084ea478d8b1456a9e071c1c2916bb7d P 40dabdafed9140fe9787acd08ba6e484 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40dabdafed9140fe9787acd08ba6e484, State: Initialized, Role: FOLLOWER
I20260619 16:52:17.737412 16273 raft_consensus.cc:348] T 084ea478d8b1456a9e071c1c2916bb7d P 46b3ce200da840748de5cffca4a76fd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40dabdafed9140fe9787acd08ba6e484" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39447 } } peers { permanent_uuid: "46b3ce200da840748de5cffca4a76fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "6f7a800440a14ac9a5a0cf1baa974e02" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34985 } }
I20260619 16:52:17.737578 16273 raft_consensus.cc:374] T 084ea478d8b1456a9e071c1c2916bb7d P 46b3ce200da840748de5cffca4a76fd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:52:17.737620 16273 raft_consensus.cc:729] T 084ea478d8b1456a9e071c1c2916bb7d P 46b3ce200da840748de5cffca4a76fd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46b3ce200da840748de5cffca4a76fd0, State: Initialized, Role: FOLLOWER
I20260619 16:52:17.737731 16275 consensus_queue.cc:260] T 084ea478d8b1456a9e071c1c2916bb7d P 40dabdafed9140fe9787acd08ba6e484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40dabdafed9140fe9787acd08ba6e484" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39447 } } peers { permanent_uuid: "46b3ce200da840748de5cffca4a76fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "6f7a800440a14ac9a5a0cf1baa974e02" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34985 } }
I20260619 16:52:17.738021 16275 ts_tablet_manager.cc:1428] T 084ea478d8b1456a9e071c1c2916bb7d P 40dabdafed9140fe9787acd08ba6e484: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 16:52:17.738039 16256 heartbeater.cc:503] Master 127.0.98.62:41581 was elected leader, sending a full tablet report...
I20260619 16:52:17.738306 16273 consensus_queue.cc:260] T 084ea478d8b1456a9e071c1c2916bb7d P 46b3ce200da840748de5cffca4a76fd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40dabdafed9140fe9787acd08ba6e484" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39447 } } peers { permanent_uuid: "46b3ce200da840748de5cffca4a76fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "6f7a800440a14ac9a5a0cf1baa974e02" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34985 } }
I20260619 16:52:17.738622 16273 ts_tablet_manager.cc:1428] T 084ea478d8b1456a9e071c1c2916bb7d P 46b3ce200da840748de5cffca4a76fd0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 16:52:17.738682 16122 heartbeater.cc:503] Master 127.0.98.62:41581 was elected leader, sending a full tablet report...
W20260619 16:52:17.756711 16123 tablet.cc:2369] T 084ea478d8b1456a9e071c1c2916bb7d P 46b3ce200da840748de5cffca4a76fd0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:52:17.809687 15989 tablet.cc:2369] T 084ea478d8b1456a9e071c1c2916bb7d P 6f7a800440a14ac9a5a0cf1baa974e02: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:52:17.841908 16281 raft_consensus.cc:482] T 084ea478d8b1456a9e071c1c2916bb7d P 46b3ce200da840748de5cffca4a76fd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:52:17.842082 16281 raft_consensus.cc:504] T 084ea478d8b1456a9e071c1c2916bb7d P 46b3ce200da840748de5cffca4a76fd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40dabdafed9140fe9787acd08ba6e484" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39447 } } peers { permanent_uuid: "46b3ce200da840748de5cffca4a76fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "6f7a800440a14ac9a5a0cf1baa974e02" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34985 } }
I20260619 16:52:17.842547 16281 leader_election.cc:288] T 084ea478d8b1456a9e071c1c2916bb7d P 46b3ce200da840748de5cffca4a76fd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40dabdafed9140fe9787acd08ba6e484 (127.0.98.3:39447), 6f7a800440a14ac9a5a0cf1baa974e02 (127.0.98.1:34985)
I20260619 16:52:17.845459 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.842890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49831 (local address 127.0.98.1:34985)
0619 16:52:17.842936 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:17.842938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:17.842949 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:52:17.842984 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:17.842985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:17.842997 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:17.843053 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:52:17.843225 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.843753 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.844743 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.845123 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.845185 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:17.845197 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:17.845339 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 16:52:17.845380 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 16:52:17.845463 16124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.842752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34985 (local address 127.0.98.2:49831)
0619 16:52:17.842815 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:17.842829 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:17.842887 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:17.843068 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:17.843069 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:17.843071 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:52:17.843159 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:17.843161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.843834 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.843835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:17.844645 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:17.844647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.845194 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.845194 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:17.845276 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:17.845280 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:52:17.845325 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 16:52:17.845374 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":16}
I20260619 16:52:17.845563 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.842880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52845 (local address 127.0.98.3:39447)
0619 16:52:17.842930 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:17.842932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:17.843056 (+ 124us) server_negotiation.cc:407] Connection header received
0619 16:52:17.843093 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:17.843094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:17.843125 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:17.843174 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:52:17.843311 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.843788 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.844738 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.845177 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.845238 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:17.845248 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:17.845372 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 16:52:17.845482 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 16:52:17.845590 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.842752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39447 (local address 127.0.98.2:52845)
0619 16:52:17.842995 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:17.843002 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:17.843048 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:17.843184 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:17.843185 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:17.843186 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:52:17.843257 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:17.843259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.843872 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.843873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:17.844651 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:17.844653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.845248 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.845249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:17.845314 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:17.845317 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:52:17.845361 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 16:52:17.845471 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":51,"thread_start_us":114,"threads_started":1}
I20260619 16:52:17.845942 15942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "084ea478d8b1456a9e071c1c2916bb7d" candidate_uuid: "46b3ce200da840748de5cffca4a76fd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f7a800440a14ac9a5a0cf1baa974e02" is_pre_election: true
I20260619 16:52:17.846118 15942 raft_consensus.cc:2454] T 084ea478d8b1456a9e071c1c2916bb7d P 6f7a800440a14ac9a5a0cf1baa974e02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46b3ce200da840748de5cffca4a76fd0 in term 0.
I20260619 16:52:17.846158 16210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "084ea478d8b1456a9e071c1c2916bb7d" candidate_uuid: "46b3ce200da840748de5cffca4a76fd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40dabdafed9140fe9787acd08ba6e484" is_pre_election: true
I20260619 16:52:17.846318 16280 raft_consensus.cc:482] T 084ea478d8b1456a9e071c1c2916bb7d P 40dabdafed9140fe9787acd08ba6e484 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:52:17.846405 16280 raft_consensus.cc:504] T 084ea478d8b1456a9e071c1c2916bb7d P 40dabdafed9140fe9787acd08ba6e484 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40dabdafed9140fe9787acd08ba6e484" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39447 } } peers { permanent_uuid: "46b3ce200da840748de5cffca4a76fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "6f7a800440a14ac9a5a0cf1baa974e02" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34985 } }
I20260619 16:52:17.846449 16010 leader_election.cc:302] T 084ea478d8b1456a9e071c1c2916bb7d P 46b3ce200da840748de5cffca4a76fd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46b3ce200da840748de5cffca4a76fd0, 6f7a800440a14ac9a5a0cf1baa974e02; no voters:
I20260619 16:52:17.846547 16210 raft_consensus.cc:2454] T 084ea478d8b1456a9e071c1c2916bb7d P 40dabdafed9140fe9787acd08ba6e484 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46b3ce200da840748de5cffca4a76fd0 in term 0.
I20260619 16:52:17.846607 16281 raft_consensus.cc:2781] T 084ea478d8b1456a9e071c1c2916bb7d P 46b3ce200da840748de5cffca4a76fd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:52:17.846653 16281 raft_consensus.cc:482] T 084ea478d8b1456a9e071c1c2916bb7d P 46b3ce200da840748de5cffca4a76fd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:52:17.846673 16281 raft_consensus.cc:3037] T 084ea478d8b1456a9e071c1c2916bb7d P 46b3ce200da840748de5cffca4a76fd0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:52:17.846745 16280 leader_election.cc:288] T 084ea478d8b1456a9e071c1c2916bb7d P 40dabdafed9140fe9787acd08ba6e484 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46b3ce200da840748de5cffca4a76fd0 (127.0.98.2:45561), 6f7a800440a14ac9a5a0cf1baa974e02 (127.0.98.1:34985)
I20260619 16:52:17.847818 16281 raft_consensus.cc:504] T 084ea478d8b1456a9e071c1c2916bb7d P 46b3ce200da840748de5cffca4a76fd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40dabdafed9140fe9787acd08ba6e484" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39447 } } peers { permanent_uuid: "46b3ce200da840748de5cffca4a76fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "6f7a800440a14ac9a5a0cf1baa974e02" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34985 } }
I20260619 16:52:17.847970 16281 leader_election.cc:288] T 084ea478d8b1456a9e071c1c2916bb7d P 46b3ce200da840748de5cffca4a76fd0 [CANDIDATE]: Term 1 election: Requested vote from peers 40dabdafed9140fe9787acd08ba6e484 (127.0.98.3:39447), 6f7a800440a14ac9a5a0cf1baa974e02 (127.0.98.1:34985)
I20260619 16:52:17.848161 16210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "084ea478d8b1456a9e071c1c2916bb7d" candidate_uuid: "46b3ce200da840748de5cffca4a76fd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40dabdafed9140fe9787acd08ba6e484"
I20260619 16:52:17.848191 15942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "084ea478d8b1456a9e071c1c2916bb7d" candidate_uuid: "46b3ce200da840748de5cffca4a76fd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f7a800440a14ac9a5a0cf1baa974e02"
I20260619 16:52:17.848248 16210 raft_consensus.cc:3037] T 084ea478d8b1456a9e071c1c2916bb7d P 40dabdafed9140fe9787acd08ba6e484 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:52:17.848280 15942 raft_consensus.cc:3037] T 084ea478d8b1456a9e071c1c2916bb7d P 6f7a800440a14ac9a5a0cf1baa974e02 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:52:17.849260 16210 raft_consensus.cc:2454] T 084ea478d8b1456a9e071c1c2916bb7d P 40dabdafed9140fe9787acd08ba6e484 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46b3ce200da840748de5cffca4a76fd0 in term 1.
I20260619 16:52:17.849270 15942 raft_consensus.cc:2454] T 084ea478d8b1456a9e071c1c2916bb7d P 6f7a800440a14ac9a5a0cf1baa974e02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46b3ce200da840748de5cffca4a76fd0 in term 1.
I20260619 16:52:17.849540 16012 leader_election.cc:302] T 084ea478d8b1456a9e071c1c2916bb7d P 46b3ce200da840748de5cffca4a76fd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40dabdafed9140fe9787acd08ba6e484, 46b3ce200da840748de5cffca4a76fd0; no voters:
I20260619 16:52:17.849669 16281 raft_consensus.cc:2781] T 084ea478d8b1456a9e071c1c2916bb7d P 46b3ce200da840748de5cffca4a76fd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:52:17.849843 16281 raft_consensus.cc:686] T 084ea478d8b1456a9e071c1c2916bb7d P 46b3ce200da840748de5cffca4a76fd0 [term 1 LEADER]: Becoming Leader. State: Replica: 46b3ce200da840748de5cffca4a76fd0, State: Running, Role: LEADER
I20260619 16:52:17.849998 16281 consensus_queue.cc:237] T 084ea478d8b1456a9e071c1c2916bb7d P 46b3ce200da840748de5cffca4a76fd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40dabdafed9140fe9787acd08ba6e484" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39447 } } peers { permanent_uuid: "46b3ce200da840748de5cffca4a76fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "6f7a800440a14ac9a5a0cf1baa974e02" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34985 } }
I20260619 16:52:17.850431 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.847151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46963 (local address 127.0.98.2:45561)
0619 16:52:17.847201 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:17.847203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:17.847211 (+ 8us) server_negotiation.cc:407] Connection header received
0619 16:52:17.847234 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:17.847235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:17.847245 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:17.847302 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:52:17.847456 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.848105 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.849451 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.850023 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.850086 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:17.850098 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:17.850334 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 16:52:17.850376 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 16:52:17.850425 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.846967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45561 (local address 127.0.98.3:46963)
0619 16:52:17.847019 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:17.847031 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:17.847069 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:17.847325 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:17.847326 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:17.847328 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:52:17.847403 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:17.847404 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.848193 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.848194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:17.849364 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:17.849366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.850128 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.850129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:17.850240 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:17.850245 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:52:17.850294 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 16:52:17.850342 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 16:52:17.850713 16076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "084ea478d8b1456a9e071c1c2916bb7d" candidate_uuid: "40dabdafed9140fe9787acd08ba6e484" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46b3ce200da840748de5cffca4a76fd0" is_pre_election: true
I20260619 16:52:17.851177 16283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.847092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34985 (local address 127.0.98.3:39581)
0619 16:52:17.847262 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:17.847271 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:17.847345 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:17.848133 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:17.848134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:17.848137 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:52:17.848214 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:17.848215 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.849133 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.849134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:17.850240 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:17.850242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.850929 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.850930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:17.851006 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:17.851011 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:52:17.851056 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 16:52:17.851101 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":121,"threads_started":1}
I20260619 16:52:17.851370 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.847151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39581 (local address 127.0.98.1:34985)
0619 16:52:17.847282 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:17.847285 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:17.847325 (+ 40us) server_negotiation.cc:407] Connection header received
0619 16:52:17.847398 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:17.847399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:17.847408 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:17.847455 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:52:17.848293 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.849028 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.850305 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.850804 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.850873 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:17.850885 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:17.851271 (+ 386us) server_negotiation.cc:299] Negotiation successful
0619 16:52:17.851318 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260619 16:52:17.851423 15796 catalog_manager.cc:5495] T 084ea478d8b1456a9e071c1c2916bb7d P 46b3ce200da840748de5cffca4a76fd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46b3ce200da840748de5cffca4a76fd0 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "46b3ce200da840748de5cffca4a76fd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40dabdafed9140fe9787acd08ba6e484" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46b3ce200da840748de5cffca4a76fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f7a800440a14ac9a5a0cf1baa974e02" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34985 } health_report { overall_health: UNKNOWN } } }
I20260619 16:52:17.851539 15942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "084ea478d8b1456a9e071c1c2916bb7d" candidate_uuid: "40dabdafed9140fe9787acd08ba6e484" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f7a800440a14ac9a5a0cf1baa974e02" is_pre_election: true
I20260619 16:52:17.851621 15942 raft_consensus.cc:2379] T 084ea478d8b1456a9e071c1c2916bb7d P 6f7a800440a14ac9a5a0cf1baa974e02 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 40dabdafed9140fe9787acd08ba6e484 in current term 1: Already voted for candidate 46b3ce200da840748de5cffca4a76fd0 in this term.
I20260619 16:52:17.851877 16144 leader_election.cc:302] T 084ea478d8b1456a9e071c1c2916bb7d P 40dabdafed9140fe9787acd08ba6e484 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40dabdafed9140fe9787acd08ba6e484; no voters: 46b3ce200da840748de5cffca4a76fd0, 6f7a800440a14ac9a5a0cf1baa974e02
I20260619 16:52:17.852072 16280 raft_consensus.cc:2726] T 084ea478d8b1456a9e071c1c2916bb7d P 40dabdafed9140fe9787acd08ba6e484 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 16:52:17.873940 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.871444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54070 (local address 127.0.98.2:45561)
0619 16:52:17.871505 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:17.871507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:17.871530 (+ 23us) server_negotiation.cc:407] Connection header received
0619 16:52:17.871577 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:17.871578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:17.871605 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:17.871673 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:52:17.871811 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.872361 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.873139 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:17.873340 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:17.873400 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:17.873550 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:52:17.873689 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:52:17.873769 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:17.873788 (+ 19us) server_negotiation.cc:299] Negotiation successful
0619 16:52:17.873815 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 16:52:17.876139 15942 raft_consensus.cc:1261] T 084ea478d8b1456a9e071c1c2916bb7d P 6f7a800440a14ac9a5a0cf1baa974e02 [term 1 FOLLOWER]: Refusing update from remote peer 46b3ce200da840748de5cffca4a76fd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:52:17.876176 16210 raft_consensus.cc:1261] T 084ea478d8b1456a9e071c1c2916bb7d P 40dabdafed9140fe9787acd08ba6e484 [term 1 FOLLOWER]: Refusing update from remote peer 46b3ce200da840748de5cffca4a76fd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:52:17.876595 16284 consensus_queue.cc:1035] T 084ea478d8b1456a9e071c1c2916bb7d P 46b3ce200da840748de5cffca4a76fd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f7a800440a14ac9a5a0cf1baa974e02" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:52:17.876765 16281 consensus_queue.cc:1035] T 084ea478d8b1456a9e071c1c2916bb7d P 46b3ce200da840748de5cffca4a76fd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40dabdafed9140fe9787acd08ba6e484" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:52:17.879884 16287 mvcc.cc:205] Tried to move back new op lower bound from 7298612993537785856 to 7298612993434513408. Current Snapshot: MvccSnapshot[applied={T|T < 7298612993537785856}]
I20260619 16:52:17.880897 16288 mvcc.cc:205] Tried to move back new op lower bound from 7298612993537785856 to 7298612993434513408. Current Snapshot: MvccSnapshot[applied={T|T < 7298612993537785856}]
I20260619 16:52:17.881069 16289 mvcc.cc:205] Tried to move back new op lower bound from 7298612993537785856 to 7298612993434513408. Current Snapshot: MvccSnapshot[applied={T|T < 7298612993537785856 or (T in {7298612993537785856})}]
I20260619 16:52:17.882815 15796 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:52:17.882936 15796 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:52:17.883975 15796 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450: Sending DeleteTablet for 3 replicas of tablet 084ea478d8b1456a9e071c1c2916bb7d
I20260619 16:52:17.884362 15922 tablet_service.cc:1514] Processing DeleteTablet for tablet 084ea478d8b1456a9e071c1c2916bb7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:52:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53830
I20260619 16:52:17.884382 16190 tablet_service.cc:1514] Processing DeleteTablet for tablet 084ea478d8b1456a9e071c1c2916bb7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:52:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49054
I20260619 16:52:17.884580 16056 tablet_service.cc:1514] Processing DeleteTablet for tablet 084ea478d8b1456a9e071c1c2916bb7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:52:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54068
I20260619 16:52:17.884811 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 15859
I20260619 16:52:17.884865 16299 tablet_replica.cc:330] T 084ea478d8b1456a9e071c1c2916bb7d P 46b3ce200da840748de5cffca4a76fd0: stopping tablet replica
I20260619 16:52:17.885398 16299 raft_consensus.cc:2229] T 084ea478d8b1456a9e071c1c2916bb7d P 46b3ce200da840748de5cffca4a76fd0 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:52:17.885562 16299 raft_consensus.cc:2258] T 084ea478d8b1456a9e071c1c2916bb7d P 46b3ce200da840748de5cffca4a76fd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:52:17.886119 16299 ts_tablet_manager.cc:1905] T 084ea478d8b1456a9e071c1c2916bb7d P 46b3ce200da840748de5cffca4a76fd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:52:17.888088 16299 ts_tablet_manager.cc:1918] T 084ea478d8b1456a9e071c1c2916bb7d P 46b3ce200da840748de5cffca4a76fd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:52:17.888206 16299 log.cc:1196] T 084ea478d8b1456a9e071c1c2916bb7d P 46b3ce200da840748de5cffca4a76fd0: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/wal/wals/084ea478d8b1456a9e071c1c2916bb7d
I20260619 16:52:17.888515 16299 ts_tablet_manager.cc:1939] T 084ea478d8b1456a9e071c1c2916bb7d P 46b3ce200da840748de5cffca4a76fd0: Deleting consensus metadata
I20260619 16:52:17.888841 16297 tablet_replica.cc:330] T 084ea478d8b1456a9e071c1c2916bb7d P 40dabdafed9140fe9787acd08ba6e484: stopping tablet replica
I20260619 16:52:17.888998 16297 raft_consensus.cc:2229] T 084ea478d8b1456a9e071c1c2916bb7d P 40dabdafed9140fe9787acd08ba6e484 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:52:17.889138 16297 raft_consensus.cc:2258] T 084ea478d8b1456a9e071c1c2916bb7d P 40dabdafed9140fe9787acd08ba6e484 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:52:17.889113 15783 catalog_manager.cc:4841] TS 46b3ce200da840748de5cffca4a76fd0 (127.0.98.2:45561): tablet 084ea478d8b1456a9e071c1c2916bb7d (table test-table [id=ec3f9ae58d294e9e8fe6edce21c6012a]) successfully deleted
I20260619 16:52:17.889670 16297 ts_tablet_manager.cc:1905] T 084ea478d8b1456a9e071c1c2916bb7d P 40dabdafed9140fe9787acd08ba6e484: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:52:17.891634 16297 ts_tablet_manager.cc:1918] T 084ea478d8b1456a9e071c1c2916bb7d P 40dabdafed9140fe9787acd08ba6e484: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:52:17.891770 16297 log.cc:1196] T 084ea478d8b1456a9e071c1c2916bb7d P 40dabdafed9140fe9787acd08ba6e484: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/wal/wals/084ea478d8b1456a9e071c1c2916bb7d
I20260619 16:52:17.892370 16297 ts_tablet_manager.cc:1939] T 084ea478d8b1456a9e071c1c2916bb7d P 40dabdafed9140fe9787acd08ba6e484: Deleting consensus metadata
I20260619 16:52:17.892942 15784 catalog_manager.cc:4841] TS 40dabdafed9140fe9787acd08ba6e484 (127.0.98.3:39447): tablet 084ea478d8b1456a9e071c1c2916bb7d (table test-table [id=ec3f9ae58d294e9e8fe6edce21c6012a]) successfully deleted
W20260619 16:52:17.893680 15782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34985 (error 108)
I20260619 16:52:17.894320 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:17.894143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34985 (local address 127.0.0.1:53836)
0619 16:52:17.894230 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:17.894275 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260619 16:52:17.894348 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 15993
W20260619 16:52:17.895668 15782 catalog_manager.cc:4584] TS 6f7a800440a14ac9a5a0cf1baa974e02 (127.0.98.1:34985): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 084ea478d8b1456a9e071c1c2916bb7d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34985: connect: Connection refused (error 111)
I20260619 16:52:17.899662 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 16127
I20260619 16:52:17.904229 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 15765
I20260619 16:52:37.909642 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41581
--webserver_interface=127.0.98.62
--webserver_port=35829
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:52:38.012064 16301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:38.012334 16301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:38.012362 16301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:38.013664 16301 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 16:52:38.013705 16301 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 16:52:38.013717 16301 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 16:52:38.013728 16301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:52:38.013741 16301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:38.013752 16301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:38.013767 16301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:52:38.013777 16301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:52:38.015416 16301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:52:38.015672 16301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:38.015929 16301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:38.025578 16308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:38.025578 16307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:38.025592 16310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:38.025640 16301 server_base.cc:1034] running on GCE node
I20260619 16:52:38.026302 16301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:38.026852 16301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:38.028059 16301 hybrid_clock.cc:648] HybridClock initialized: now 1781887958028026 us; error 64 us; skew 500 ppm
Jun 19 16:52:38 dist-test-slave-jsw9 krb5kdc[15752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:38.031327 16301 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:52:38.031922 16301 webserver.cc:460] Webserver started at http://127.0.98.62:35829/ using document root <none> and password file <none>
I20260619 16:52:38.032203 16301 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:38.032279 16301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:38.034024 16301 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:52:38.034976 16317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:38.035166 16301 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:52:38.035266 16301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "d81eda6159264f73977473029f411450"
format_stamp: "Formatted at 2026-06-19 16:52:17 on dist-test-slave-jsw9"
server_key: "89fccf1af9c4e617c7e3b2083adcd4c2"
server_key_iv: "1307d26a6a5e9351f5ffa16780311094"
server_key_version: "encryptionkey@0"
I20260619 16:52:38.035825 16301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:38.045049 16301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:38.047497 16301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:38.047650 16301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:38.052791 16301 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41581
I20260619 16:52:38.052876 16369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41581 every 8 connection(s)
I20260619 16:52:38.053476 16301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:52:38.055510 16370 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:52:38.056455 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 16301
I20260619 16:52:38.056479 16370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450: Bootstrap starting.
I20260619 16:52:38.056780 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34985
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41581
--webserver_port=36105
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:52:38.067711 16370 log.cc:824] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450: Log is configured to *not* fsync() on all Append() calls
I20260619 16:52:38.070232 16370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:52:38.070547 16370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450: Bootstrap complete.
I20260619 16:52:38.073016 16370 raft_consensus.cc:348] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d81eda6159264f73977473029f411450" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41581 } }
I20260619 16:52:38.073305 16370 raft_consensus.cc:729] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d81eda6159264f73977473029f411450, State: Initialized, Role: FOLLOWER
I20260619 16:52:38.073484 16370 consensus_queue.cc:260] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d81eda6159264f73977473029f411450" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41581 } }
I20260619 16:52:38.073566 16370 raft_consensus.cc:388] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:52:38.073626 16370 raft_consensus.cc:482] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:52:38.073680 16370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:52:38.074950 16370 raft_consensus.cc:504] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d81eda6159264f73977473029f411450" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41581 } }
I20260619 16:52:38.075073 16370 leader_election.cc:302] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d81eda6159264f73977473029f411450; no voters:
I20260619 16:52:38.075268 16370 leader_election.cc:288] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 16:52:38.075325 16374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 16:52:38.075487 16374 raft_consensus.cc:686] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [term 2 LEADER]: Becoming Leader. State: Replica: d81eda6159264f73977473029f411450, State: Running, Role: LEADER
I20260619 16:52:38.075596 16370 sys_catalog.cc:564] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:52:38.075618 16374 consensus_queue.cc:237] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d81eda6159264f73977473029f411450" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41581 } }
I20260619 16:52:38.075999 16375 sys_catalog.cc:455] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d81eda6159264f73977473029f411450" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d81eda6159264f73977473029f411450" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41581 } } }
I20260619 16:52:38.076079 16375 sys_catalog.cc:458] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [sys.catalog]: This master's current role is: LEADER
I20260619 16:52:38.076424 16376 sys_catalog.cc:455] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d81eda6159264f73977473029f411450. Latest consensus state: current_term: 2 leader_uuid: "d81eda6159264f73977473029f411450" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d81eda6159264f73977473029f411450" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41581 } } }
I20260619 16:52:38.076517 16376 sys_catalog.cc:458] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [sys.catalog]: This master's current role is: LEADER
I20260619 16:52:38.076638 16379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:52:38.077716 16379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:52:38.077880 16379 catalog_manager.cc:1259] Loaded cluster ID: 13220e2f0c844b62beb7fd4c6d13a2ad
I20260619 16:52:38.078145 16379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:52:38.079439 16379 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:52:38.079680 16379 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450: Loaded TSK: 0
I20260619 16:52:38.088814 16379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450: Generated new TSK 1
I20260619 16:52:38.088977 16379 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 16:52:38.152453 16372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:38.152742 16372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:38.152781 16372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:38.154070 16372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:38.154120 16372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:38.154158 16372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:52:38.155777 16372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34985
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36105
--tserver_master_addrs=127.0.98.62:41581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:52:38.156064 16372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:38.156322 16372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:38.162078 16398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:38.162076 16397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:38.162109 16400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:38.162618 16372 server_base.cc:1034] running on GCE node
I20260619 16:52:38.162868 16372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:38.163149 16372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:38.164306 16372 hybrid_clock.cc:648] HybridClock initialized: now 1781887958164291 us; error 34 us; skew 500 ppm
Jun 19 16:52:38 dist-test-slave-jsw9 krb5kdc[15752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:38.167402 16372 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:52:38.167975 16372 webserver.cc:460] Webserver started at http://127.0.98.1:36105/ using document root <none> and password file <none>
I20260619 16:52:38.168242 16372 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:38.168314 16372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:38.169817 16372 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 16:52:38.170526 16407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:38.170742 16372 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:52:38.170861 16372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/wal
uuid: "6f7a800440a14ac9a5a0cf1baa974e02"
format_stamp: "Formatted at 2026-06-19 16:52:17 on dist-test-slave-jsw9"
server_key: "26d313b6c765283e576cb1d772b2df45"
server_key_iv: "2e64450e2059689a096657a292862996"
server_key_version: "encryptionkey@0"
I20260619 16:52:38.171499 16372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:38.184612 16372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:38.187894 16372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:38.188093 16372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:38.188407 16372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:52:38.188995 16414 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 16:52:38.190794 16372 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 16:52:38.190865 16372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 16:52:38.190908 16372 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 16:52:38.192468 16372 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 16:52:38.192528 16372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 16:52:38.192655 16414 tablet_bootstrap.cc:492] T 084ea478d8b1456a9e071c1c2916bb7d P 6f7a800440a14ac9a5a0cf1baa974e02: Bootstrap starting.
I20260619 16:52:38.201062 16372 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34985
I20260619 16:52:38.201406 16521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34985 every 8 connection(s)
I20260619 16:52:38.201614 16372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:52:38.203497 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 16372
I20260619 16:52:38.203814 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45561
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41581
--webserver_port=44573
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:52:38.206663 16414 log.cc:824] T 084ea478d8b1456a9e071c1c2916bb7d P 6f7a800440a14ac9a5a0cf1baa974e02: Log is configured to *not* fsync() on all Append() calls
I20260619 16:52:38.210310 16414 tablet_bootstrap.cc:492] T 084ea478d8b1456a9e071c1c2916bb7d P 6f7a800440a14ac9a5a0cf1baa974e02: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:52:38.210714 16414 tablet_bootstrap.cc:492] T 084ea478d8b1456a9e071c1c2916bb7d P 6f7a800440a14ac9a5a0cf1baa974e02: Bootstrap complete.
I20260619 16:52:38.211956 16414 ts_tablet_manager.cc:1397] T 084ea478d8b1456a9e071c1c2916bb7d P 6f7a800440a14ac9a5a0cf1baa974e02: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.004s
I20260619 16:52:38.214401 16414 raft_consensus.cc:348] T 084ea478d8b1456a9e071c1c2916bb7d P 6f7a800440a14ac9a5a0cf1baa974e02 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40dabdafed9140fe9787acd08ba6e484" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39447 } } peers { permanent_uuid: "46b3ce200da840748de5cffca4a76fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "6f7a800440a14ac9a5a0cf1baa974e02" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34985 } }
I20260619 16:52:38.214818 16414 raft_consensus.cc:729] T 084ea478d8b1456a9e071c1c2916bb7d P 6f7a800440a14ac9a5a0cf1baa974e02 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f7a800440a14ac9a5a0cf1baa974e02, State: Initialized, Role: FOLLOWER
I20260619 16:52:38.215000 16414 consensus_queue.cc:260] T 084ea478d8b1456a9e071c1c2916bb7d P 6f7a800440a14ac9a5a0cf1baa974e02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40dabdafed9140fe9787acd08ba6e484" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39447 } } peers { permanent_uuid: "46b3ce200da840748de5cffca4a76fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "6f7a800440a14ac9a5a0cf1baa974e02" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34985 } }
I20260619 16:52:38.215402 16414 ts_tablet_manager.cc:1428] T 084ea478d8b1456a9e071c1c2916bb7d P 6f7a800440a14ac9a5a0cf1baa974e02: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
Jun 19 16:52:38 dist-test-slave-jsw9 krb5kdc[15752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:52:38.225539 16525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:38.203102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39721 (local address 127.0.98.62:41581)
0619 16:52:38.210512 (+ 7410us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:38.210526 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:38.210595 (+ 69us) server_negotiation.cc:407] Connection header received
0619 16:52:38.210823 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:38.210832 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:38.210908 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:38.212273 (+ 1365us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:38.212381 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.213895 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.215668 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.216404 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.216566 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:38.220745 (+ 4179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:38.220753 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:38.220753 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:38.220772 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:38.223148 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:38.223832 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:38.223836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:38.223837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:38.223991 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:38.224305 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:38.224305 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:38.224306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:38.224673 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:38.224772 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:38.225126 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 16:52:38.225293 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7213,"thread_start_us":98,"threads_started":1}
I20260619 16:52:38.225543 16524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:38.202783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.1:39721)
0619 16:52:38.203011 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:38.203107 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:38.204440 (+ 1333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:38.211196 (+ 6756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:38.211203 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:38.211649 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:52:38.212008 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:38.212018 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.214069 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.214071 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:38.215413 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:38.215417 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.216573 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.216575 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:38.216736 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:38.217546 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:52:38.217566 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:52:38.220437 (+ 2871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:52:38.223293 (+ 2856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:38.223307 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:38.223311 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:38.223716 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:38.224070 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:38.224071 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:38.224072 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:38.224235 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:38.224748 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:52:38.224754 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:52:38.224941 (+ 187us) client_negotiation.cc:769] Sending connection context
0619 16:52:38.225059 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 16:52:38.225293 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":108,"threads_started":1}
I20260619 16:52:38.226830 16522 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41581
I20260619 16:52:38.227048 16522 heartbeater.cc:464] Registering TS with master...
I20260619 16:52:38.227320 16522 heartbeater.cc:511] Master 127.0.98.62:41581 requested a full tablet report, sending...
I20260619 16:52:38.228308 16334 ts_manager.cc:194] Registered new tserver with Master: 6f7a800440a14ac9a5a0cf1baa974e02 (127.0.98.1:34985)
I20260619 16:52:38.229990 16334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39721
I20260619 16:52:38.231462 16522 heartbeater.cc:503] Master 127.0.98.62:41581 was elected leader, sending a full tablet report...
Jun 19 16:52:38 dist-test-slave-jsw9 krb5kdc[15752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 16:52:38.248030 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:38.229525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34985 (local address 127.0.0.1:49166)
0619 16:52:38.230439 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:38.230487 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:38.230574 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:38.231088 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:38.231095 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:38.231737 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:52:38.231833 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:38.231838 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.233405 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.233406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:38.234643 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:38.234648 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.236153 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.236162 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:38.236416 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:38.240058 (+ 3642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:52:38.240067 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:52:38.243685 (+ 3618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:52:38.246129 (+ 2444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:38.246144 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:38.246146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:38.246469 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:38.246792 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:38.246793 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:38.246794 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:38.246867 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:38.247187 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:52:38.247198 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:52:38.247274 (+ 76us) client_negotiation.cc:769] Sending connection context
0619 16:52:38.247482 (+ 208us) client_negotiation.cc:240] Negotiation successful
0619 16:52:38.247903 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"thread_start_us":179,"threads_started":1}
I20260619 16:52:38.248030 16530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:38.229934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49166 (local address 127.0.98.1:34985)
0619 16:52:38.230884 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:38.230890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:38.230903 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:52:38.230947 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:38.230950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:38.230972 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:38.231932 (+ 960us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:38.232008 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.233281 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.234829 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.235440 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.236092 (+ 652us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:38.243844 (+ 7752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:38.243860 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:38.243861 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:38.243872 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:38.245969 (+ 2097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:38.246634 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:38.246640 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:38.246641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:38.246716 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:38.246930 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:38.246931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:38.246931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:38.247122 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:38.247202 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:38.247760 (+ 558us) server_negotiation.cc:299] Negotiation successful
0619 16:52:38.247899 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":908,"thread_start_us":112,"threads_started":1}
I20260619 16:52:38.248811 16456 tablet_service.cc:1514] Processing DeleteTablet for tablet 084ea478d8b1456a9e071c1c2916bb7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:52:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49166
I20260619 16:52:38.249550 16531 tablet_replica.cc:330] stopping tablet replica
I20260619 16:52:38.249821 16531 raft_consensus.cc:2229] T 084ea478d8b1456a9e071c1c2916bb7d P 6f7a800440a14ac9a5a0cf1baa974e02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:52:38.249928 16531 raft_consensus.cc:2258] T 084ea478d8b1456a9e071c1c2916bb7d P 6f7a800440a14ac9a5a0cf1baa974e02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:52:38.251380 16531 ts_tablet_manager.cc:1905] T 084ea478d8b1456a9e071c1c2916bb7d P 6f7a800440a14ac9a5a0cf1baa974e02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:52:38.257505 16531 ts_tablet_manager.cc:1918] T 084ea478d8b1456a9e071c1c2916bb7d P 6f7a800440a14ac9a5a0cf1baa974e02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:52:38.257652 16531 log.cc:1196] T 084ea478d8b1456a9e071c1c2916bb7d P 6f7a800440a14ac9a5a0cf1baa974e02: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/wal/wals/084ea478d8b1456a9e071c1c2916bb7d
I20260619 16:52:38.257922 16531 ts_tablet_manager.cc:1939] T 084ea478d8b1456a9e071c1c2916bb7d P 6f7a800440a14ac9a5a0cf1baa974e02: Deleting consensus metadata
I20260619 16:52:38.258879 16319 catalog_manager.cc:4841] TS 6f7a800440a14ac9a5a0cf1baa974e02 (127.0.98.1:34985): tablet 084ea478d8b1456a9e071c1c2916bb7d (table test-table [id=ec3f9ae58d294e9e8fe6edce21c6012a]) successfully deleted
W20260619 16:52:38.320684 16526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:38.320964 16526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:38.321002 16526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:38.322863 16526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:38.322938 16526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:38.322985 16526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:52:38.324788 16526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45561
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44573
--tserver_master_addrs=127.0.98.62:41581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:52:38.325112 16526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:38.325383 16526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:38.331444 16537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:38.331425 16536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:38.331429 16539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:38.331879 16526 server_base.cc:1034] running on GCE node
I20260619 16:52:38.332113 16526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:38.332398 16526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:38.333562 16526 hybrid_clock.cc:648] HybridClock initialized: now 1781887958333550 us; error 33 us; skew 500 ppm
Jun 19 16:52:38 dist-test-slave-jsw9 krb5kdc[15752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:38.336284 16526 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:52:38.336840 16526 webserver.cc:460] Webserver started at http://127.0.98.2:44573/ using document root <none> and password file <none>
I20260619 16:52:38.337071 16526 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:38.337139 16526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:38.338670 16526 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:52:38.339398 16546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:38.339581 16526 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:52:38.339668 16526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/wal
uuid: "46b3ce200da840748de5cffca4a76fd0"
format_stamp: "Formatted at 2026-06-19 16:52:17 on dist-test-slave-jsw9"
server_key: "a7c927e12c01e5230609af3ae41aefb3"
server_key_iv: "5c63f9b0c2237301d74a5d2183a026d5"
server_key_version: "encryptionkey@0"
I20260619 16:52:38.340075 16526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:38.364158 16526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:38.366683 16526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:38.366834 16526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:38.367129 16526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:52:38.367475 16526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:52:38.367516 16526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:38.367545 16526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:52:38.367558 16526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:38.374694 16526 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45561
I20260619 16:52:38.374747 16659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45561 every 8 connection(s)
I20260619 16:52:38.375181 16526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 16:52:38 dist-test-slave-jsw9 krb5kdc[15752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:52:38.382654 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 16526
I20260619 16:52:38.382995 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39447
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41581
--webserver_port=36917
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:52:38.386166 16525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:38.375980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38177 (local address 127.0.98.62:41581)
0619 16:52:38.376038 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:38.376041 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:38.376309 (+ 268us) server_negotiation.cc:407] Connection header received
0619 16:52:38.376918 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:38.376920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:38.376932 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:38.376983 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:38.378079 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.378704 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.379842 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.380070 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.380144 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:38.382681 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:38.382685 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:38.382685 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:38.382696 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:38.384292 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:38.385064 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:38.385065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:38.385066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:38.385140 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:38.385397 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:38.385398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:38.385398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:38.385573 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:38.385637 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:38.385941 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 16:52:38.386058 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 16:52:38.386430 16662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:38.375971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.2:38177)
0619 16:52:38.376150 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:38.376218 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:38.376825 (+ 607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:38.377051 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:38.377055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:38.377711 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:52:38.377952 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:38.377957 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.378813 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.378814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:38.379740 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:38.379743 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.380149 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.380150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:38.380279 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:38.380861 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:52:38.380871 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:52:38.382531 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:52:38.384413 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:38.384424 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:38.384427 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:38.384932 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:38.385203 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:38.385204 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:38.385205 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:38.385335 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:38.385635 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:52:38.385643 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:52:38.385810 (+ 167us) client_negotiation.cc:769] Sending connection context
0619 16:52:38.385911 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 16:52:38.386129 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":103,"threads_started":1}
I20260619 16:52:38.386971 16660 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41581
I20260619 16:52:38.387086 16660 heartbeater.cc:464] Registering TS with master...
I20260619 16:52:38.387297 16660 heartbeater.cc:511] Master 127.0.98.62:41581 requested a full tablet report, sending...
I20260619 16:52:38.387746 16334 ts_manager.cc:194] Registered new tserver with Master: 46b3ce200da840748de5cffca4a76fd0 (127.0.98.2:45561)
I20260619 16:52:38.388319 16334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38177
W20260619 16:52:38.479029 16663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:38.479337 16663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:38.479368 16663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:38.480742 16663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:38.480801 16663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:38.480832 16663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:52:38.482514 16663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39447
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36917
--tserver_master_addrs=127.0.98.62:41581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:52:38.482761 16663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:38.482990 16663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:38.488536 16669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:38.488574 16668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:38.488595 16663 server_base.cc:1034] running on GCE node
W20260619 16:52:38.488560 16671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:38.489017 16663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:38.489311 16663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:38.490456 16663 hybrid_clock.cc:648] HybridClock initialized: now 1781887958490443 us; error 30 us; skew 500 ppm
Jun 19 16:52:38 dist-test-slave-jsw9 krb5kdc[15752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:38.493249 16663 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:52:38.493755 16663 webserver.cc:460] Webserver started at http://127.0.98.3:36917/ using document root <none> and password file <none>
I20260619 16:52:38.493978 16663 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:38.494048 16663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:38.495628 16663 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:52:38.496276 16678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:38.496490 16663 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:52:38.496591 16663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/wal
uuid: "40dabdafed9140fe9787acd08ba6e484"
format_stamp: "Formatted at 2026-06-19 16:52:17 on dist-test-slave-jsw9"
server_key: "f025c98b58d1b72e43696426d199f462"
server_key_iv: "b963d1252c920a56cbc7bfebc7768b4e"
server_key_version: "encryptionkey@0"
I20260619 16:52:38.497028 16663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:38.528833 16663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:38.531222 16663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:38.531378 16663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:38.531672 16663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:52:38.532018 16663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:52:38.532054 16663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:38.532088 16663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:52:38.532111 16663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:38.538856 16663 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39447
I20260619 16:52:38.538906 16791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39447 every 8 connection(s)
I20260619 16:52:38.539368 16663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:52:38.540187 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 16663
Jun 19 16:52:38 dist-test-slave-jsw9 krb5kdc[15752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:52:38.548365 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:38.540665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43466 (local address 127.0.98.62:41581)
0619 16:52:38.540779 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:38.540780 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:38.540908 (+ 128us) server_negotiation.cc:407] Connection header received
0619 16:52:38.541000 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:38.541000 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:38.541012 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:38.541086 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:38.541909 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.542545 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.543458 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.543677 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.543728 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:38.545254 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:38.545256 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:38.545256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:38.545264 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:38.547038 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:38.547435 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:38.547436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:38.547437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:38.547521 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:38.547743 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:38.547744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:38.547744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:38.547888 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:38.548107 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:38.548146 (+ 39us) server_negotiation.cc:299] Negotiation successful
0619 16:52:38.548250 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":61,"threads_started":1}
I20260619 16:52:38.550666 16525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:38.540266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52075 (local address 127.0.98.62:41581)
0619 16:52:38.540454 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:38.540456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:38.540753 (+ 297us) server_negotiation.cc:407] Connection header received
0619 16:52:38.541506 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:38.541507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:38.541517 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:38.541581 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:38.542791 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.543345 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.544724 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.544930 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.545016 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:38.547610 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:38.547612 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:38.547612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:38.547622 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:38.549168 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:38.549646 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:38.549647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:38.549647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:38.549704 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:38.549939 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:38.549940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:38.549940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:38.550065 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:38.550121 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:38.550474 (+ 353us) server_negotiation.cc:299] Negotiation successful
0619 16:52:38.550563 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260619 16:52:38.550937 16794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:38.540287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.3:52075)
0619 16:52:38.540486 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:38.540542 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:38.541346 (+ 804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:38.541672 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:38.541677 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:38.542428 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:52:38.542701 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:38.542708 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.543430 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.543432 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:38.544613 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:38.544617 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.545047 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.545048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:38.545209 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:38.545785 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:52:38.545795 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:52:38.547515 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:52:38.549245 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:38.549254 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:38.549257 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:38.549569 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:38.549758 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:52:38.549759 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:52:38.549759 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:52:38.549883 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:52:38.550120 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:52:38.550128 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:52:38.550342 (+ 214us) client_negotiation.cc:769] Sending connection context
0619 16:52:38.550446 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 16:52:38.550668 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":94,"threads_started":1}
I20260619 16:52:38.551514 16792 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41581
I20260619 16:52:38.551614 16792 heartbeater.cc:464] Registering TS with master...
I20260619 16:52:38.551795 16792 heartbeater.cc:511] Master 127.0.98.62:41581 requested a full tablet report, sending...
I20260619 16:52:38.552147 16334 ts_manager.cc:194] Registered new tserver with Master: 40dabdafed9140fe9787acd08ba6e484 (127.0.98.3:39447)
I20260619 16:52:38.552511 16334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52075
I20260619 16:52:38.560266 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:52:38.563392 16525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:38.560711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43474 (local address 127.0.98.62:41581)
0619 16:52:38.560762 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:38.560764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:38.560885 (+ 121us) server_negotiation.cc:407] Connection header received
0619 16:52:38.560959 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:38.560959 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:38.561028 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:38.561107 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:52:38.561244 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.561764 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.562612 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.562822 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.562891 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:38.563093 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:52:38.563152 (+ 59us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 16:52:38.563322 (+ 170us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43474: token expired
Metrics: {"server-negotiator.queue_time_us":24}
W20260619 16:52:38.563462 16525 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43474: token expired
W20260619 16:52:38.563642 16261 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:41581)
I20260619 16:52:38.571419 16525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:38.563926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43484 (local address 127.0.98.62:41581)
0619 16:52:38.563973 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:38.563974 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:38.563983 (+ 9us) server_negotiation.cc:407] Connection header received
0619 16:52:38.564017 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:38.564018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:38.564025 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:38.564068 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:52:38.564557 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.565337 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.566255 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.566452 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.566520 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:38.568433 (+ 1913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:52:38.568436 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:52:38.568436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:52:38.568452 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:52:38.570460 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:38.570826 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:38.570827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:38.570827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:38.570881 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:52:38.571005 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:52:38.571005 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:52:38.571005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:52:38.571114 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:52:38.571195 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:38.571276 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 16:52:38.571339 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 16:52:38.572808 16334 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43484:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:52:38.574956 16334 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:52:38.577459 16456 tablet_service.cc:1467] Processing CreateTablet for tablet 6da69207858246f1b46effd2f41589c7 (DEFAULT_TABLE table=test-table [id=74f034c0a0cb4d9e9530db24c65bc054]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:52:38.577719 16456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6da69207858246f1b46effd2f41589c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:52:38.579408 16414 tablet_bootstrap.cc:492] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02: Bootstrap starting.
I20260619 16:52:38.580001 16414 tablet_bootstrap.cc:654] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02: Neither blocks nor log segments found. Creating new log.
I20260619 16:52:38.581624 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:38.576976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45561 (local address 127.0.0.1:48214)
0619 16:52:38.577033 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:38.577045 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:38.577093 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:38.577523 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:38.577524 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:38.577526 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:52:38.577592 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:38.577594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.578905 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.578906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:38.580108 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:38.580111 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.581082 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.581083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:38.581227 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:38.581256 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 16:52:38.581339 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 16:52:38.581484 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":43}
I20260619 16:52:38.581636 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:38.577122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48214 (local address 127.0.98.2:45561)
0619 16:52:38.577324 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:38.577330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:38.577341 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:52:38.577391 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:38.577393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:38.577410 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:38.577491 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:52:38.577662 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.578814 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.580207 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.580984 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.581061 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:38.581137 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:38.581472 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 16:52:38.581528 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":104,"threads_started":1}
I20260619 16:52:38.582679 16594 tablet_service.cc:1467] Processing CreateTablet for tablet 6da69207858246f1b46effd2f41589c7 (DEFAULT_TABLE table=test-table [id=74f034c0a0cb4d9e9530db24c65bc054]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:52:38.583118 16594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6da69207858246f1b46effd2f41589c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:52:38.583946 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:38.576976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39447 (local address 127.0.0.1:52858)
0619 16:52:38.577276 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:38.577286 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:38.577352 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:38.580064 (+ 2712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:38.580065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:38.580068 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:52:38.580114 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:38.580115 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.581317 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.581318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:38.582425 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:38.582429 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.583346 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.583347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:38.583441 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:38.583446 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:52:38.583761 (+ 315us) client_negotiation.cc:240] Negotiation successful
0619 16:52:38.583869 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":175,"thread_start_us":68,"threads_started":1}
I20260619 16:52:38.584148 16800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:38.579454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52858 (local address 127.0.98.3:39447)
0619 16:52:38.579796 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:38.579802 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:38.579815 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:52:38.579860 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:38.579862 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:38.579880 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:38.579960 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:52:38.580190 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.581241 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.582507 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.583262 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.583331 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:38.583392 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:38.583635 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 16:52:38.583680 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":110,"threads_started":1}
I20260619 16:52:38.584326 16414 tablet_bootstrap.cc:492] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02: No bootstrap required, opened a new log
I20260619 16:52:38.584375 16414 ts_tablet_manager.cc:1397] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260619 16:52:38.584794 16414 raft_consensus.cc:348] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46b3ce200da840748de5cffca4a76fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "40dabdafed9140fe9787acd08ba6e484" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39447 } } peers { permanent_uuid: "6f7a800440a14ac9a5a0cf1baa974e02" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34985 } }
I20260619 16:52:38.584877 16414 raft_consensus.cc:374] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:52:38.584937 16414 raft_consensus.cc:729] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f7a800440a14ac9a5a0cf1baa974e02, State: Initialized, Role: FOLLOWER
I20260619 16:52:38.584988 16414 consensus_queue.cc:260] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46b3ce200da840748de5cffca4a76fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "40dabdafed9140fe9787acd08ba6e484" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39447 } } peers { permanent_uuid: "6f7a800440a14ac9a5a0cf1baa974e02" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34985 } }
I20260619 16:52:38.585091 16414 ts_tablet_manager.cc:1428] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:52:38.585878 16726 tablet_service.cc:1467] Processing CreateTablet for tablet 6da69207858246f1b46effd2f41589c7 (DEFAULT_TABLE table=test-table [id=74f034c0a0cb4d9e9530db24c65bc054]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:52:38.586313 16726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6da69207858246f1b46effd2f41589c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:52:38.589926 16803 tablet_bootstrap.cc:492] T 6da69207858246f1b46effd2f41589c7 P 40dabdafed9140fe9787acd08ba6e484: Bootstrap starting.
I20260619 16:52:38.590701 16803 tablet_bootstrap.cc:654] T 6da69207858246f1b46effd2f41589c7 P 40dabdafed9140fe9787acd08ba6e484: Neither blocks nor log segments found. Creating new log.
I20260619 16:52:38.591187 16803 log.cc:824] T 6da69207858246f1b46effd2f41589c7 P 40dabdafed9140fe9787acd08ba6e484: Log is configured to *not* fsync() on all Append() calls
I20260619 16:52:38.591885 16803 tablet_bootstrap.cc:492] T 6da69207858246f1b46effd2f41589c7 P 40dabdafed9140fe9787acd08ba6e484: No bootstrap required, opened a new log
I20260619 16:52:38.591936 16803 ts_tablet_manager.cc:1397] T 6da69207858246f1b46effd2f41589c7 P 40dabdafed9140fe9787acd08ba6e484: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:52:38.593416 16803 raft_consensus.cc:348] T 6da69207858246f1b46effd2f41589c7 P 40dabdafed9140fe9787acd08ba6e484 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46b3ce200da840748de5cffca4a76fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "40dabdafed9140fe9787acd08ba6e484" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39447 } } peers { permanent_uuid: "6f7a800440a14ac9a5a0cf1baa974e02" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34985 } }
I20260619 16:52:38.593604 16803 raft_consensus.cc:374] T 6da69207858246f1b46effd2f41589c7 P 40dabdafed9140fe9787acd08ba6e484 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:52:38.593629 16803 raft_consensus.cc:729] T 6da69207858246f1b46effd2f41589c7 P 40dabdafed9140fe9787acd08ba6e484 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40dabdafed9140fe9787acd08ba6e484, State: Initialized, Role: FOLLOWER
I20260619 16:52:38.593761 16803 consensus_queue.cc:260] T 6da69207858246f1b46effd2f41589c7 P 40dabdafed9140fe9787acd08ba6e484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46b3ce200da840748de5cffca4a76fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "40dabdafed9140fe9787acd08ba6e484" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39447 } } peers { permanent_uuid: "6f7a800440a14ac9a5a0cf1baa974e02" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34985 } }
I20260619 16:52:38.594084 16803 ts_tablet_manager.cc:1428] T 6da69207858246f1b46effd2f41589c7 P 40dabdafed9140fe9787acd08ba6e484: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:52:38.594108 16792 heartbeater.cc:503] Master 127.0.98.62:41581 was elected leader, sending a full tablet report...
I20260619 16:52:38.596441 16802 tablet_bootstrap.cc:492] T 6da69207858246f1b46effd2f41589c7 P 46b3ce200da840748de5cffca4a76fd0: Bootstrap starting.
I20260619 16:52:38.597210 16802 tablet_bootstrap.cc:654] T 6da69207858246f1b46effd2f41589c7 P 46b3ce200da840748de5cffca4a76fd0: Neither blocks nor log segments found. Creating new log.
I20260619 16:52:38.597854 16802 log.cc:824] T 6da69207858246f1b46effd2f41589c7 P 46b3ce200da840748de5cffca4a76fd0: Log is configured to *not* fsync() on all Append() calls
I20260619 16:52:38.598801 16802 tablet_bootstrap.cc:492] T 6da69207858246f1b46effd2f41589c7 P 46b3ce200da840748de5cffca4a76fd0: No bootstrap required, opened a new log
I20260619 16:52:38.598872 16802 ts_tablet_manager.cc:1397] T 6da69207858246f1b46effd2f41589c7 P 46b3ce200da840748de5cffca4a76fd0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:52:38.600569 16802 raft_consensus.cc:348] T 6da69207858246f1b46effd2f41589c7 P 46b3ce200da840748de5cffca4a76fd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46b3ce200da840748de5cffca4a76fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "40dabdafed9140fe9787acd08ba6e484" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39447 } } peers { permanent_uuid: "6f7a800440a14ac9a5a0cf1baa974e02" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34985 } }
I20260619 16:52:38.600739 16802 raft_consensus.cc:374] T 6da69207858246f1b46effd2f41589c7 P 46b3ce200da840748de5cffca4a76fd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:52:38.600772 16802 raft_consensus.cc:729] T 6da69207858246f1b46effd2f41589c7 P 46b3ce200da840748de5cffca4a76fd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46b3ce200da840748de5cffca4a76fd0, State: Initialized, Role: FOLLOWER
I20260619 16:52:38.600901 16802 consensus_queue.cc:260] T 6da69207858246f1b46effd2f41589c7 P 46b3ce200da840748de5cffca4a76fd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46b3ce200da840748de5cffca4a76fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "40dabdafed9140fe9787acd08ba6e484" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39447 } } peers { permanent_uuid: "6f7a800440a14ac9a5a0cf1baa974e02" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34985 } }
I20260619 16:52:38.601228 16802 ts_tablet_manager.cc:1428] T 6da69207858246f1b46effd2f41589c7 P 46b3ce200da840748de5cffca4a76fd0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 16:52:38.601269 16660 heartbeater.cc:503] Master 127.0.98.62:41581 was elected leader, sending a full tablet report...
W20260619 16:52:38.625893 16661 tablet.cc:2369] T 6da69207858246f1b46effd2f41589c7 P 46b3ce200da840748de5cffca4a76fd0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:52:38.703056 16523 tablet.cc:2369] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:52:38.790086 16793 tablet.cc:2369] T 6da69207858246f1b46effd2f41589c7 P 40dabdafed9140fe9787acd08ba6e484: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:52:38.815888 16528 raft_consensus.cc:482] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:52:38.816099 16528 raft_consensus.cc:504] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46b3ce200da840748de5cffca4a76fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "40dabdafed9140fe9787acd08ba6e484" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39447 } } peers { permanent_uuid: "6f7a800440a14ac9a5a0cf1baa974e02" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34985 } }
I20260619 16:52:38.816550 16528 leader_election.cc:288] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46b3ce200da840748de5cffca4a76fd0 (127.0.98.2:45561), 40dabdafed9140fe9787acd08ba6e484 (127.0.98.3:39447)
I20260619 16:52:38.819900 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:38.816857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41527 (local address 127.0.98.2:45561)
0619 16:52:38.816909 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:38.816912 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:38.817047 (+ 135us) server_negotiation.cc:407] Connection header received
0619 16:52:38.817270 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:38.817271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:38.817286 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:38.817340 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:52:38.817505 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.818026 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.819104 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.819532 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.819594 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:38.819607 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:38.819774 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 16:52:38.819822 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 16:52:38.819900 16800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:38.816927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55807 (local address 127.0.98.3:39447)
0619 16:52:38.816979 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:38.816981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:38.817070 (+ 89us) server_negotiation.cc:407] Connection header received
0619 16:52:38.817171 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:38.817173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:38.817190 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:38.817240 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:52:38.817518 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.818026 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.819101 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.819523 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.819594 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:38.819607 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:38.819768 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 16:52:38.819816 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 16:52:38.819916 16808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:38.816777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45561 (local address 127.0.98.1:41527)
0619 16:52:38.816969 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:38.816982 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:38.817186 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:38.817349 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:38.817350 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:38.817352 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:52:38.817428 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:38.817431 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.818111 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.818112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:38.819009 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:38.819011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.819613 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.819614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:38.819687 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:38.819706 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 16:52:38.819752 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 16:52:38.819810 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":3,"thread_start_us":92,"threads_started":1}
I20260619 16:52:38.819923 16809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:38.816807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39447 (local address 127.0.98.1:55807)
0619 16:52:38.816974 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:38.816984 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:38.817074 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:38.817275 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:52:38.817276 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:52:38.817278 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:52:38.817448 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:38.817451 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.818114 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.818115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:38.819013 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:52:38.819015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.819600 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.819601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:52:38.819682 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:38.819704 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 16:52:38.819752 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 16:52:38.819810 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":94,"thread_start_us":107,"threads_started":1}
I20260619 16:52:38.820304 16614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6da69207858246f1b46effd2f41589c7" candidate_uuid: "6f7a800440a14ac9a5a0cf1baa974e02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46b3ce200da840748de5cffca4a76fd0" is_pre_election: true
I20260619 16:52:38.820483 16614 raft_consensus.cc:2454] T 6da69207858246f1b46effd2f41589c7 P 46b3ce200da840748de5cffca4a76fd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f7a800440a14ac9a5a0cf1baa974e02 in term 0.
I20260619 16:52:38.820737 16410 leader_election.cc:302] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46b3ce200da840748de5cffca4a76fd0, 6f7a800440a14ac9a5a0cf1baa974e02; no voters:
I20260619 16:52:38.820881 16528 raft_consensus.cc:2781] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:52:38.820962 16528 raft_consensus.cc:482] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:52:38.820983 16528 raft_consensus.cc:3037] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:52:38.820930 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6da69207858246f1b46effd2f41589c7" candidate_uuid: "6f7a800440a14ac9a5a0cf1baa974e02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40dabdafed9140fe9787acd08ba6e484" is_pre_election: true
I20260619 16:52:38.821089 16746 raft_consensus.cc:2454] T 6da69207858246f1b46effd2f41589c7 P 40dabdafed9140fe9787acd08ba6e484 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f7a800440a14ac9a5a0cf1baa974e02 in term 0.
I20260619 16:52:38.821858 16528 raft_consensus.cc:504] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46b3ce200da840748de5cffca4a76fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "40dabdafed9140fe9787acd08ba6e484" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39447 } } peers { permanent_uuid: "6f7a800440a14ac9a5a0cf1baa974e02" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34985 } }
I20260619 16:52:38.821991 16528 leader_election.cc:288] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [CANDIDATE]: Term 1 election: Requested vote from peers 46b3ce200da840748de5cffca4a76fd0 (127.0.98.2:45561), 40dabdafed9140fe9787acd08ba6e484 (127.0.98.3:39447)
I20260619 16:52:38.822232 16746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6da69207858246f1b46effd2f41589c7" candidate_uuid: "6f7a800440a14ac9a5a0cf1baa974e02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40dabdafed9140fe9787acd08ba6e484"
I20260619 16:52:38.822333 16746 raft_consensus.cc:3037] T 6da69207858246f1b46effd2f41589c7 P 40dabdafed9140fe9787acd08ba6e484 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:52:38.822355 16614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6da69207858246f1b46effd2f41589c7" candidate_uuid: "6f7a800440a14ac9a5a0cf1baa974e02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46b3ce200da840748de5cffca4a76fd0"
I20260619 16:52:38.822453 16614 raft_consensus.cc:3037] T 6da69207858246f1b46effd2f41589c7 P 46b3ce200da840748de5cffca4a76fd0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:52:38.823203 16746 raft_consensus.cc:2454] T 6da69207858246f1b46effd2f41589c7 P 40dabdafed9140fe9787acd08ba6e484 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f7a800440a14ac9a5a0cf1baa974e02 in term 1.
I20260619 16:52:38.823235 16614 raft_consensus.cc:2454] T 6da69207858246f1b46effd2f41589c7 P 46b3ce200da840748de5cffca4a76fd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f7a800440a14ac9a5a0cf1baa974e02 in term 1.
I20260619 16:52:38.823374 16410 leader_election.cc:302] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46b3ce200da840748de5cffca4a76fd0, 6f7a800440a14ac9a5a0cf1baa974e02; no voters:
I20260619 16:52:38.823488 16528 raft_consensus.cc:2781] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:52:38.823606 16528 raft_consensus.cc:686] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [term 1 LEADER]: Becoming Leader. State: Replica: 6f7a800440a14ac9a5a0cf1baa974e02, State: Running, Role: LEADER
I20260619 16:52:38.823673 16528 consensus_queue.cc:237] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46b3ce200da840748de5cffca4a76fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "40dabdafed9140fe9787acd08ba6e484" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39447 } } peers { permanent_uuid: "6f7a800440a14ac9a5a0cf1baa974e02" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34985 } }
I20260619 16:52:38.824580 16334 catalog_manager.cc:5495] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f7a800440a14ac9a5a0cf1baa974e02 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6f7a800440a14ac9a5a0cf1baa974e02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46b3ce200da840748de5cffca4a76fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40dabdafed9140fe9787acd08ba6e484" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f7a800440a14ac9a5a0cf1baa974e02" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34985 } health_report { overall_health: HEALTHY } } }
I20260619 16:52:38.878997 16813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:38.876226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49178 (local address 127.0.98.1:34985)
0619 16:52:38.876433 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 16:52:38.876435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:52:38.876443 (+ 8us) server_negotiation.cc:407] Connection header received
0619 16:52:38.876524 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:52:38.876525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:52:38.876553 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:52:38.876643 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:52:38.876798 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.877341 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.878122 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:52:38.878378 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:52:38.878481 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:52:38.878628 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:52:38.878771 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:52:38.878829 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 16:52:38.878874 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 16:52:38.878895 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":130,"threads_started":1}
I20260619 16:52:38.881049 16746 raft_consensus.cc:1261] T 6da69207858246f1b46effd2f41589c7 P 40dabdafed9140fe9787acd08ba6e484 [term 1 FOLLOWER]: Refusing update from remote peer 6f7a800440a14ac9a5a0cf1baa974e02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:52:38.881049 16614 raft_consensus.cc:1261] T 6da69207858246f1b46effd2f41589c7 P 46b3ce200da840748de5cffca4a76fd0 [term 1 FOLLOWER]: Refusing update from remote peer 6f7a800440a14ac9a5a0cf1baa974e02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:52:38.881413 16528 consensus_queue.cc:1035] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46b3ce200da840748de5cffca4a76fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:52:38.881603 16810 consensus_queue.cc:1035] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40dabdafed9140fe9787acd08ba6e484" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:52:38.885032 16816 mvcc.cc:205] Tried to move back new op lower bound from 7298613079573676032 to 7298613079342436352. Current Snapshot: MvccSnapshot[applied={T|T < 7298613079573676032}]
I20260619 16:52:38.886114 16815 mvcc.cc:205] Tried to move back new op lower bound from 7298613079573676032 to 7298613079342436352. Current Snapshot: MvccSnapshot[applied={T|T < 7298613079573676032 or (T in {7298613079573676032})}]
I20260619 16:52:38.887840 16333 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:52:38.887952 16333 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:52:38.888792 16333 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450: Sending DeleteTablet for 3 replicas of tablet 6da69207858246f1b46effd2f41589c7
I20260619 16:52:38.889108 16456 tablet_service.cc:1514] Processing DeleteTablet for tablet 6da69207858246f1b46effd2f41589c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:52:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49166
I20260619 16:52:38.889138 16594 tablet_service.cc:1514] Processing DeleteTablet for tablet 6da69207858246f1b46effd2f41589c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:52:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48214
I20260619 16:52:38.889375 16823 tablet_replica.cc:330] T 6da69207858246f1b46effd2f41589c7 P 46b3ce200da840748de5cffca4a76fd0: stopping tablet replica
I20260619 16:52:38.889504 16823 raft_consensus.cc:2229] T 6da69207858246f1b46effd2f41589c7 P 46b3ce200da840748de5cffca4a76fd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:52:38.889535 16726 tablet_service.cc:1514] Processing DeleteTablet for tablet 6da69207858246f1b46effd2f41589c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:52:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52858
I20260619 16:52:38.889636 16823 raft_consensus.cc:2258] T 6da69207858246f1b46effd2f41589c7 P 46b3ce200da840748de5cffca4a76fd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:52:38.889816 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 16372
I20260619 16:52:38.889833 16824 tablet_replica.cc:330] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02: stopping tablet replica
I20260619 16:52:38.889882 16824 raft_consensus.cc:2229] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:52:38.889986 16824 raft_consensus.cc:2258] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:52:38.890978 16825 tablet_replica.cc:330] T 6da69207858246f1b46effd2f41589c7 P 40dabdafed9140fe9787acd08ba6e484: stopping tablet replica
I20260619 16:52:38.891079 16825 raft_consensus.cc:2229] T 6da69207858246f1b46effd2f41589c7 P 40dabdafed9140fe9787acd08ba6e484 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:52:38.891180 16825 raft_consensus.cc:2258] T 6da69207858246f1b46effd2f41589c7 P 40dabdafed9140fe9787acd08ba6e484 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:52:38.891520 16823 ts_tablet_manager.cc:1905] T 6da69207858246f1b46effd2f41589c7 P 46b3ce200da840748de5cffca4a76fd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:52:38.891741 16825 ts_tablet_manager.cc:1905] T 6da69207858246f1b46effd2f41589c7 P 40dabdafed9140fe9787acd08ba6e484: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:52:38.893745 16823 ts_tablet_manager.cc:1918] T 6da69207858246f1b46effd2f41589c7 P 46b3ce200da840748de5cffca4a76fd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:52:38.893759 16825 ts_tablet_manager.cc:1918] T 6da69207858246f1b46effd2f41589c7 P 40dabdafed9140fe9787acd08ba6e484: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:52:38.893836 16823 log.cc:1196] T 6da69207858246f1b46effd2f41589c7 P 46b3ce200da840748de5cffca4a76fd0: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/wal/wals/6da69207858246f1b46effd2f41589c7
I20260619 16:52:38.893862 16825 log.cc:1196] T 6da69207858246f1b46effd2f41589c7 P 40dabdafed9140fe9787acd08ba6e484: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/wal/wals/6da69207858246f1b46effd2f41589c7
I20260619 16:52:38.894083 16823 ts_tablet_manager.cc:1939] T 6da69207858246f1b46effd2f41589c7 P 46b3ce200da840748de5cffca4a76fd0: Deleting consensus metadata
I20260619 16:52:38.894258 16825 ts_tablet_manager.cc:1939] T 6da69207858246f1b46effd2f41589c7 P 40dabdafed9140fe9787acd08ba6e484: Deleting consensus metadata
I20260619 16:52:38.894663 16320 catalog_manager.cc:4841] TS 46b3ce200da840748de5cffca4a76fd0 (127.0.98.2:45561): tablet 6da69207858246f1b46effd2f41589c7 (table test-table [id=74f034c0a0cb4d9e9530db24c65bc054]) successfully deleted
I20260619 16:52:38.894757 16321 catalog_manager.cc:4841] TS 40dabdafed9140fe9787acd08ba6e484 (127.0.98.3:39447): tablet 6da69207858246f1b46effd2f41589c7 (table test-table [id=74f034c0a0cb4d9e9530db24c65bc054]) successfully deleted
W20260619 16:52:38.896350 16319 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34985 (error 108)
I20260619 16:52:38.896889 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:38.896764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34985 (local address 127.0.0.1:49182)
0619 16:52:38.896812 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:38.896852 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260619 16:52:38.896994 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 16526
W20260619 16:52:38.897009 16319 catalog_manager.cc:4584] TS 6f7a800440a14ac9a5a0cf1baa974e02 (127.0.98.1:34985): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6da69207858246f1b46effd2f41589c7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34985: connect: Connection refused (error 111)
I20260619 16:52:38.903076 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 16663
I20260619 16:52:38.907927 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 16301
I20260619 16:52:58.913390 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41581
--webserver_interface=127.0.98.62
--webserver_port=35829
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:52:59.010594 16827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:59.010881 16827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:59.010910 16827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:59.012360 16827 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 16:52:59.012421 16827 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 16:52:59.012434 16827 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 16:52:59.012444 16827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:52:59.012457 16827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:59.012470 16827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:59.012480 16827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:52:59.012491 16827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:52:59.014307 16827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:52:59.014621 16827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:59.014899 16827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:59.020750 16832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:59.020808 16827 server_base.cc:1034] running on GCE node
W20260619 16:52:59.020768 16835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:59.020771 16833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:59.021255 16827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:59.021636 16827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:59.022794 16827 hybrid_clock.cc:648] HybridClock initialized: now 1781887979022776 us; error 40 us; skew 500 ppm
Jun 19 16:52:59 dist-test-slave-jsw9 krb5kdc[15752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:59.025938 16827 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:52:59.026532 16827 webserver.cc:460] Webserver started at http://127.0.98.62:35829/ using document root <none> and password file <none>
I20260619 16:52:59.026808 16827 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:59.026857 16827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:59.028376 16827 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 16:52:59.029084 16842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:59.029278 16827 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 16:52:59.029369 16827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "d81eda6159264f73977473029f411450"
format_stamp: "Formatted at 2026-06-19 16:52:17 on dist-test-slave-jsw9"
server_key: "89fccf1af9c4e617c7e3b2083adcd4c2"
server_key_iv: "1307d26a6a5e9351f5ffa16780311094"
server_key_version: "encryptionkey@0"
I20260619 16:52:59.029827 16827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:59.039381 16827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:59.042145 16827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:59.042622 16827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:59.047469 16827 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41581
I20260619 16:52:59.047506 16894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41581 every 8 connection(s)
I20260619 16:52:59.047936 16827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:52:59.049659 16895 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:52:59.050065 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 16827
I20260619 16:52:59.050422 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34985
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41581
--webserver_port=36105
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:52:59.050652 16895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450: Bootstrap starting.
I20260619 16:52:59.063102 16895 log.cc:824] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450: Log is configured to *not* fsync() on all Append() calls
I20260619 16:52:59.065930 16895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:52:59.066309 16895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450: Bootstrap complete.
I20260619 16:52:59.068722 16895 raft_consensus.cc:348] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d81eda6159264f73977473029f411450" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41581 } }
I20260619 16:52:59.068975 16895 raft_consensus.cc:729] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d81eda6159264f73977473029f411450, State: Initialized, Role: FOLLOWER
I20260619 16:52:59.069145 16895 consensus_queue.cc:260] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d81eda6159264f73977473029f411450" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41581 } }
I20260619 16:52:59.069214 16895 raft_consensus.cc:388] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:52:59.069259 16895 raft_consensus.cc:482] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:52:59.069308 16895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [term 2 FOLLOWER]: Advancing to term 3
I20260619 16:52:59.070533 16895 raft_consensus.cc:504] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d81eda6159264f73977473029f411450" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41581 } }
I20260619 16:52:59.070636 16895 leader_election.cc:302] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d81eda6159264f73977473029f411450; no voters:
I20260619 16:52:59.070848 16895 leader_election.cc:288] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 16:52:59.070912 16899 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 16:52:59.071051 16899 raft_consensus.cc:686] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [term 3 LEADER]: Becoming Leader. State: Replica: d81eda6159264f73977473029f411450, State: Running, Role: LEADER
I20260619 16:52:59.071156 16895 sys_catalog.cc:564] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:52:59.071146 16899 consensus_queue.cc:237] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d81eda6159264f73977473029f411450" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41581 } }
I20260619 16:52:59.071488 16899 sys_catalog.cc:455] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d81eda6159264f73977473029f411450. Latest consensus state: current_term: 3 leader_uuid: "d81eda6159264f73977473029f411450" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d81eda6159264f73977473029f411450" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41581 } } }
I20260619 16:52:59.071493 16901 sys_catalog.cc:455] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d81eda6159264f73977473029f411450" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d81eda6159264f73977473029f411450" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41581 } } }
I20260619 16:52:59.071646 16901 sys_catalog.cc:458] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [sys.catalog]: This master's current role is: LEADER
I20260619 16:52:59.071574 16899 sys_catalog.cc:458] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450 [sys.catalog]: This master's current role is: LEADER
I20260619 16:52:59.072216 16905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:52:59.073179 16905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:52:59.073294 16905 catalog_manager.cc:1259] Loaded cluster ID: 13220e2f0c844b62beb7fd4c6d13a2ad
I20260619 16:52:59.073324 16905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:52:59.074522 16905 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:52:59.074807 16905 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450: Loaded TSK: 0
I20260619 16:52:59.074860 16905 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d81eda6159264f73977473029f411450: Loaded TSK: 1
F20260619 16:52:59.075146 16905 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f19003e59ad google::LogMessage::Fail() at ??:0
@ 0x7f19003e888a google::LogMessage::SendToLog() at ??:0
@ 0x7f19003e54de google::LogMessage::Flush() at ??:0
@ 0x7f19003e6a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1902b52104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f1902b4fd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f19083431cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f190834348e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f1908343594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f1908309c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f1908313ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f19022b30d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f19022a8954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f190310c6db start_thread at ??:0
@ 0x7f18feee361f clone at ??:0
W20260619 16:52:59.166934 16897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:59.167177 16897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:59.167207 16897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:59.168574 16897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:59.168627 16897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:59.168658 16897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:52:59.170348 16897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34985
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36105
--tserver_master_addrs=127.0.98.62:41581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:52:59.170665 16897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:59.170912 16897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:59.176810 16936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:59.176839 16897 server_base.cc:1034] running on GCE node
W20260619 16:52:59.176934 16939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:59.176810 16937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:59.177265 16897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:59.177556 16897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:59.178748 16897 hybrid_clock.cc:648] HybridClock initialized: now 1781887979178723 us; error 41 us; skew 500 ppm
Jun 19 16:52:59 dist-test-slave-jsw9 krb5kdc[15752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:59.182364 16897 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:52:59.182961 16897 webserver.cc:460] Webserver started at http://127.0.98.1:36105/ using document root <none> and password file <none>
I20260619 16:52:59.183208 16897 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:59.183351 16897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:59.185036 16897 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 16:52:59.185984 16946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:59.186216 16897 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:52:59.186321 16897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/wal
uuid: "6f7a800440a14ac9a5a0cf1baa974e02"
format_stamp: "Formatted at 2026-06-19 16:52:17 on dist-test-slave-jsw9"
server_key: "26d313b6c765283e576cb1d772b2df45"
server_key_iv: "2e64450e2059689a096657a292862996"
server_key_version: "encryptionkey@0"
I20260619 16:52:59.186903 16897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:59.208266 16897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:59.211333 16897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:59.211524 16897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:59.211859 16897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:52:59.212554 16953 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 16:52:59.214723 16897 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 16:52:59.214859 16897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260619 16:52:59.214905 16897 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 16:52:59.216229 16897 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 16:52:59.216302 16897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 16:52:59.216490 16953 tablet_bootstrap.cc:492] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02: Bootstrap starting.
I20260619 16:52:59.225479 16897 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34985
I20260619 16:52:59.225688 17060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34985 every 8 connection(s)
I20260619 16:52:59.226070 16897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:52:59.227814 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 16897
I20260619 16:52:59.228139 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45561
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41581
--webserver_port=44573
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:52:59.229710 16953 log.cc:824] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02: Log is configured to *not* fsync() on all Append() calls
I20260619 16:52:59.232463 16953 tablet_bootstrap.cc:492] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:52:59.232772 16953 tablet_bootstrap.cc:492] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02: Bootstrap complete.
I20260619 16:52:59.233834 16953 ts_tablet_manager.cc:1397] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.005s
I20260619 16:52:59.235548 16953 raft_consensus.cc:348] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46b3ce200da840748de5cffca4a76fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "40dabdafed9140fe9787acd08ba6e484" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39447 } } peers { permanent_uuid: "6f7a800440a14ac9a5a0cf1baa974e02" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34985 } }
I20260619 16:52:59.235919 16953 raft_consensus.cc:729] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f7a800440a14ac9a5a0cf1baa974e02, State: Initialized, Role: FOLLOWER
I20260619 16:52:59.236130 16953 consensus_queue.cc:260] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46b3ce200da840748de5cffca4a76fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "40dabdafed9140fe9787acd08ba6e484" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39447 } } peers { permanent_uuid: "6f7a800440a14ac9a5a0cf1baa974e02" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34985 } }
I20260619 16:52:59.236495 16953 ts_tablet_manager.cc:1428] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
W20260619 16:52:59.305164 17063 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:52:59.227018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.1:33649)
0619 16:52:59.227243 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:59.227301 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 16:52:59.228530 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:52:59.304836 (+ 76306us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":113,"threads_started":1}
W20260619 16:52:59.305586 17061 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41581 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41581: Client connection negotiation failed: client connection to 127.0.98.62:41581: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 16:52:59.306087 17063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:59.305909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.1:35633)
0619 16:52:59.305980 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:59.306056 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 16:52:59.306504 17063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:59.306407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.1:38383)
0619 16:52:59.306452 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:59.306481 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260619 16:52:59.306759 17061 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 16:52:59.327860 17064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:59.328145 17064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:59.328174 17064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:59.330664 17064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:59.330757 17064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:59.330796 17064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:52:59.332844 17064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45561
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44573
--tserver_master_addrs=127.0.98.62:41581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:52:59.333132 17064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:59.333384 17064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:59.339066 17074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:59.339105 17072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:59.339099 17071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:59.339550 17064 server_base.cc:1034] running on GCE node
I20260619 16:52:59.339756 17064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:59.340023 17064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:59.341229 17064 hybrid_clock.cc:648] HybridClock initialized: now 1781887979341208 us; error 38 us; skew 500 ppm
Jun 19 16:52:59 dist-test-slave-jsw9 krb5kdc[15752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:59.344090 17064 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:52:59.344652 17064 webserver.cc:460] Webserver started at http://127.0.98.2:44573/ using document root <none> and password file <none>
I20260619 16:52:59.344870 17064 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:59.344923 17064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:59.346446 17064 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:52:59.347077 17081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:59.347227 17064 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:59.347311 17064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/wal
uuid: "46b3ce200da840748de5cffca4a76fd0"
format_stamp: "Formatted at 2026-06-19 16:52:17 on dist-test-slave-jsw9"
server_key: "a7c927e12c01e5230609af3ae41aefb3"
server_key_iv: "5c63f9b0c2237301d74a5d2183a026d5"
server_key_version: "encryptionkey@0"
I20260619 16:52:59.347736 17064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:59.358664 17064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:59.361259 17064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:59.361440 17064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:59.361713 17064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:52:59.362058 17064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:52:59.362094 17064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:59.362128 17064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:52:59.362151 17064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:59.369720 17064 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45561
I20260619 16:52:59.369773 17194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45561 every 8 connection(s)
I20260619 16:52:59.370297 17064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:52:59.371587 17197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:59.371141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.2:35797)
0619 16:52:59.371315 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:59.371400 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
W20260619 16:52:59.372031 17195 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41581 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41581: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
I20260619 16:52:59.372328 17197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:59.372233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.2:57681)
0619 16:52:59.372280 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:59.372307 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 16:52:59.372614 17197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:59.372511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.2:33239)
0619 16:52:59.372558 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:59.372591 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 16:52:59.372735 17195 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 16:52:59.377249 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 17064
I20260619 16:52:59.377578 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39447
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41581
--webserver_port=36917
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:52:59.469080 17198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:52:59.469373 17198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:52:59.469404 17198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:52:59.470790 17198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:52:59.470844 17198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:52:59.470882 17198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:52:59.472541 17198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39447
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36917
--tserver_master_addrs=127.0.98.62:41581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:52:59.472815 17198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:52:59.473065 17198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:52:59.478713 17204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:59.478802 17203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:52:59.478713 17206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:52:59.479293 17198 server_base.cc:1034] running on GCE node
I20260619 16:52:59.479573 17198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:52:59.479840 17198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:52:59.481024 17198 hybrid_clock.cc:648] HybridClock initialized: now 1781887979481003 us; error 30 us; skew 500 ppm
Jun 19 16:52:59 dist-test-slave-jsw9 krb5kdc[15752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:52:59.484076 17198 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:52:59.484647 17198 webserver.cc:460] Webserver started at http://127.0.98.3:36917/ using document root <none> and password file <none>
I20260619 16:52:59.484880 17198 fs_manager.cc:359] Metadata directory not provided
I20260619 16:52:59.484927 17198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:52:59.486459 17198 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:52:59.487146 17213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:59.487389 17198 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:52:59.487493 17198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/wal
uuid: "40dabdafed9140fe9787acd08ba6e484"
format_stamp: "Formatted at 2026-06-19 16:52:17 on dist-test-slave-jsw9"
server_key: "f025c98b58d1b72e43696426d199f462"
server_key_iv: "b963d1252c920a56cbc7bfebc7768b4e"
server_key_version: "encryptionkey@0"
I20260619 16:52:59.487918 17198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:52:59.511101 17198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:52:59.513655 17198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:52:59.513804 17198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:52:59.514097 17198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:52:59.514505 17198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:52:59.514540 17198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:59.514568 17198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:52:59.514593 17198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:52:59.521507 17198 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39447
I20260619 16:52:59.521689 17326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39447 every 8 connection(s)
I20260619 16:52:59.521988 17198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:52:59.523387 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:59.522913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.3:44903)
0619 16:52:59.523125 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:59.523215 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":113,"threads_started":1}
W20260619 16:52:59.523833 17327 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41581 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41581: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
I20260619 16:52:59.524183 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:59.524046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.3:42905)
0619 16:52:59.524127 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:59.524156 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 16:52:59.524261 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 17198
I20260619 16:52:59.524748 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:52:59.524653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.3:42881)
0619 16:52:59.524698 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 16:52:59.524721 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 16:52:59.524855 17327 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 16:53:00.307643 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:00.307295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.1:44317)
0619 16:53:00.307498 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:00.307554 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":120,"threads_started":1}
I20260619 16:53:00.373471 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:00.373165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.2:53883)
0619 16:53:00.373335 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:00.373399 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":74,"threads_started":1}
I20260619 16:53:00.525707 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:00.525337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.3:33971)
0619 16:53:00.525544 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:00.525601 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":83,"threads_started":1}
I20260619 16:53:00.976104 17334 raft_consensus.cc:482] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:53:00.976260 17334 raft_consensus.cc:504] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46b3ce200da840748de5cffca4a76fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "40dabdafed9140fe9787acd08ba6e484" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39447 } } peers { permanent_uuid: "6f7a800440a14ac9a5a0cf1baa974e02" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34985 } }
I20260619 16:53:00.976806 17334 leader_election.cc:288] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46b3ce200da840748de5cffca4a76fd0 (127.0.98.2:45561), 40dabdafed9140fe9787acd08ba6e484 (127.0.98.3:39447)
Jun 19 16:53:00 dist-test-slave-jsw9 krb5kdc[15752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Jun 19 16:53:00 dist-test-slave-jsw9 krb5kdc[15752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260619 16:53:00.992915 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:00.977232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39543 (local address 127.0.98.3:39447)
0619 16:53:00.977391 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:00.977406 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:00.977448 (+ 42us) server_negotiation.cc:407] Connection header received
0619 16:53:00.978136 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:00.978139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:00.978237 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:00.978614 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:00.980533 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:00.982344 (+ 1811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:00.984379 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:00.985137 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:00.985285 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:00.988655 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:00.988663 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:00.988664 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:00.988682 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:00.990865 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:00.991415 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:00.991420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:00.991421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:00.991571 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:00.991840 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:00.991842 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:00.991842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:00.992194 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:00.992268 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:00.992618 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 16:53:00.992774 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
I20260619 16:53:00.992935 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:00.977226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53699 (local address 127.0.98.2:45561)
0619 16:53:00.977391 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:00.977406 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:00.977455 (+ 49us) server_negotiation.cc:407] Connection header received
0619 16:53:00.978130 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:00.978135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:00.978233 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:00.978854 (+ 621us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:00.980532 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:00.982344 (+ 1812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:00.984373 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:00.985136 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:00.985285 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:00.989053 (+ 3768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:00.989061 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:00.989061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:00.989083 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:00.991028 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:00.991564 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:00.991567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:00.991568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:00.991680 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:00.991879 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:00.991880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:00.991881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:00.992193 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:00.992268 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:00.992672 (+ 404us) server_negotiation.cc:299] Negotiation successful
0619 16:53:00.992815 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":55,"threads_started":1}
I20260619 16:53:00.992918 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:00.977041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45561 (local address 127.0.98.1:53699)
0619 16:53:00.977399 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:00.977407 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:00.977456 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:00.979409 (+ 1953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:00.979413 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:00.979902 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:00.980254 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:00.980266 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:00.982588 (+ 2322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:00.982589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:00.984209 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:00.984216 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:00.985297 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:00.985298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:00.985480 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:00.986274 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:53:00.986287 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:00.988839 (+ 2552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:00.991147 (+ 2308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:00.991149 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:00.991149 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:00.991415 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:00.991745 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:00.991746 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:00.991746 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:00.991809 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:00.992272 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:00.992279 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:00.992443 (+ 164us) client_negotiation.cc:769] Sending connection context
0619 16:53:00.992594 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 16:53:00.992794 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":130,"thread_start_us":80,"threads_started":1}
I20260619 16:53:00.992996 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:00.977041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39447 (local address 127.0.98.1:39543)
0619 16:53:00.977238 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:00.977252 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:00.977326 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:00.978829 (+ 1503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:00.978835 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:00.979902 (+ 1067us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:00.980254 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:00.980266 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:00.982530 (+ 2264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:00.982532 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:00.984209 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:00.984216 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:00.985296 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:00.985298 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:00.985480 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:00.986287 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:53:00.986291 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:00.988477 (+ 2186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:00.990960 (+ 2483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:00.990974 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:00.990978 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:00.991325 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:00.991658 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:00.991659 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:00.991659 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:00.991777 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:00.992270 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:00.992279 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:00.992441 (+ 162us) client_negotiation.cc:769] Sending connection context
0619 16:53:00.992542 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 16:53:00.992866 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":209,"thread_start_us":90,"threads_started":1}
W20260619 16:53:00.994233 16949 leader_election.cc:341] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 46b3ce200da840748de5cffca4a76fd0 (127.0.98.2:45561): Not found: Tablet not found: 6da69207858246f1b46effd2f41589c7
W20260619 16:53:00.994369 16950 leader_election.cc:341] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 40dabdafed9140fe9787acd08ba6e484 (127.0.98.3:39447): Not found: Tablet not found: 6da69207858246f1b46effd2f41589c7
I20260619 16:53:00.994453 16950 leader_election.cc:302] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f7a800440a14ac9a5a0cf1baa974e02; no voters: 40dabdafed9140fe9787acd08ba6e484, 46b3ce200da840748de5cffca4a76fd0
I20260619 16:53:00.994576 17334 raft_consensus.cc:2726] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:53:01.308504 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:01.308296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.1:38895)
0619 16:53:01.308385 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:01.308449 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 16:53:01.374521 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:01.374136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.2:44559)
0619 16:53:01.374337 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:01.374426 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":101,"threads_started":1}
I20260619 16:53:01.526798 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:01.526504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.3:38553)
0619 16:53:01.526675 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:01.526736 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":77,"threads_started":1}
I20260619 16:53:02.309505 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:02.309158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.1:56231)
0619 16:53:02.309384 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:02.309442 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":94,"threads_started":1}
I20260619 16:53:02.375420 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:02.375124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.2:47577)
0619 16:53:02.375295 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:02.375354 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":80,"threads_started":1}
I20260619 16:53:02.527688 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:02.527403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.3:39129)
0619 16:53:02.527578 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:02.527627 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":83,"threads_started":1}
I20260619 16:53:03.062920 17344 raft_consensus.cc:482] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:53:03.063072 17344 raft_consensus.cc:504] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46b3ce200da840748de5cffca4a76fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "40dabdafed9140fe9787acd08ba6e484" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39447 } } peers { permanent_uuid: "6f7a800440a14ac9a5a0cf1baa974e02" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34985 } }
I20260619 16:53:03.063242 17344 leader_election.cc:288] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46b3ce200da840748de5cffca4a76fd0 (127.0.98.2:45561), 40dabdafed9140fe9787acd08ba6e484 (127.0.98.3:39447)
W20260619 16:53:03.063720 16949 leader_election.cc:341] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 46b3ce200da840748de5cffca4a76fd0 (127.0.98.2:45561): Not found: Tablet not found: 6da69207858246f1b46effd2f41589c7
W20260619 16:53:03.063814 16950 leader_election.cc:341] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 40dabdafed9140fe9787acd08ba6e484 (127.0.98.3:39447): Not found: Tablet not found: 6da69207858246f1b46effd2f41589c7
I20260619 16:53:03.063887 16950 leader_election.cc:302] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f7a800440a14ac9a5a0cf1baa974e02; no voters: 40dabdafed9140fe9787acd08ba6e484, 46b3ce200da840748de5cffca4a76fd0
I20260619 16:53:03.064016 17344 raft_consensus.cc:2726] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:53:03.310428 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:03.310023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.1:57229)
0619 16:53:03.310263 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:03.310319 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":109,"threads_started":1}
I20260619 16:53:03.376379 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:03.376059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.2:45837)
0619 16:53:03.376249 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:03.376295 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":113,"threads_started":1}
I20260619 16:53:03.528666 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:03.528325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.3:51149)
0619 16:53:03.528526 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:03.528581 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":123,"threads_started":1}
I20260619 16:53:04.311355 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:04.311100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.1:35833)
0619 16:53:04.311259 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:04.311296 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":92,"threads_started":1}
I20260619 16:53:04.377346 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:04.377033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.2:58731)
0619 16:53:04.377230 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:04.377276 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":80,"threads_started":1}
I20260619 16:53:04.529542 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:04.529231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.3:60071)
0619 16:53:04.529423 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:04.529475 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":87,"threads_started":1}
I20260619 16:53:05.157011 17351 raft_consensus.cc:482] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:53:05.157133 17351 raft_consensus.cc:504] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46b3ce200da840748de5cffca4a76fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "40dabdafed9140fe9787acd08ba6e484" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39447 } } peers { permanent_uuid: "6f7a800440a14ac9a5a0cf1baa974e02" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34985 } }
I20260619 16:53:05.157334 17351 leader_election.cc:288] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46b3ce200da840748de5cffca4a76fd0 (127.0.98.2:45561), 40dabdafed9140fe9787acd08ba6e484 (127.0.98.3:39447)
W20260619 16:53:05.157750 16949 leader_election.cc:341] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 46b3ce200da840748de5cffca4a76fd0 (127.0.98.2:45561): Not found: Tablet not found: 6da69207858246f1b46effd2f41589c7
W20260619 16:53:05.157848 16950 leader_election.cc:341] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 40dabdafed9140fe9787acd08ba6e484 (127.0.98.3:39447): Not found: Tablet not found: 6da69207858246f1b46effd2f41589c7
I20260619 16:53:05.157884 16950 leader_election.cc:302] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f7a800440a14ac9a5a0cf1baa974e02; no voters: 40dabdafed9140fe9787acd08ba6e484, 46b3ce200da840748de5cffca4a76fd0
I20260619 16:53:05.157961 17351 raft_consensus.cc:2726] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:53:05.312294 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:05.311998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.1:55947)
0619 16:53:05.312172 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:05.312230 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":81,"threads_started":1}
I20260619 16:53:05.378374 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:05.377998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.2:36893)
0619 16:53:05.378236 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:05.378301 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":80,"threads_started":1}
I20260619 16:53:05.530479 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:05.530175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.3:52297)
0619 16:53:05.530360 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:05.530414 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":82,"threads_started":1}
I20260619 16:53:06.313066 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:06.312780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.1:39265)
0619 16:53:06.312957 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:06.313001 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":76,"threads_started":1}
I20260619 16:53:06.379204 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:06.378945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.2:34303)
0619 16:53:06.379096 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:06.379138 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":80,"threads_started":1}
I20260619 16:53:06.531411 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:06.531069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.3:36965)
0619 16:53:06.531254 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:06.531310 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":81,"threads_started":1}
I20260619 16:53:06.718616 17358 raft_consensus.cc:482] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:53:06.718742 17358 raft_consensus.cc:504] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46b3ce200da840748de5cffca4a76fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "40dabdafed9140fe9787acd08ba6e484" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39447 } } peers { permanent_uuid: "6f7a800440a14ac9a5a0cf1baa974e02" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34985 } }
I20260619 16:53:06.718928 17358 leader_election.cc:288] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46b3ce200da840748de5cffca4a76fd0 (127.0.98.2:45561), 40dabdafed9140fe9787acd08ba6e484 (127.0.98.3:39447)
W20260619 16:53:06.719331 16949 leader_election.cc:341] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 46b3ce200da840748de5cffca4a76fd0 (127.0.98.2:45561): Not found: Tablet not found: 6da69207858246f1b46effd2f41589c7
W20260619 16:53:06.719420 16950 leader_election.cc:341] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 40dabdafed9140fe9787acd08ba6e484 (127.0.98.3:39447): Not found: Tablet not found: 6da69207858246f1b46effd2f41589c7
I20260619 16:53:06.719521 16950 leader_election.cc:302] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f7a800440a14ac9a5a0cf1baa974e02; no voters: 40dabdafed9140fe9787acd08ba6e484, 46b3ce200da840748de5cffca4a76fd0
I20260619 16:53:06.719688 17358 raft_consensus.cc:2726] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:53:07.314107 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:07.313725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.1:32901)
0619 16:53:07.313965 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:07.314026 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":114,"threads_started":1}
I20260619 16:53:07.380224 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:07.379859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.2:39635)
0619 16:53:07.380096 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:07.380153 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":105,"threads_started":1}
I20260619 16:53:07.532373 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:07.532069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.3:41809)
0619 16:53:07.532255 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:07.532308 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":86,"threads_started":1}
I20260619 16:53:08.227015 17362 raft_consensus.cc:482] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:53:08.227128 17362 raft_consensus.cc:504] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46b3ce200da840748de5cffca4a76fd0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45561 } } peers { permanent_uuid: "40dabdafed9140fe9787acd08ba6e484" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39447 } } peers { permanent_uuid: "6f7a800440a14ac9a5a0cf1baa974e02" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34985 } }
I20260619 16:53:08.227301 17362 leader_election.cc:288] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46b3ce200da840748de5cffca4a76fd0 (127.0.98.2:45561), 40dabdafed9140fe9787acd08ba6e484 (127.0.98.3:39447)
W20260619 16:53:08.227780 16949 leader_election.cc:341] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 46b3ce200da840748de5cffca4a76fd0 (127.0.98.2:45561): Not found: Tablet not found: 6da69207858246f1b46effd2f41589c7
W20260619 16:53:08.227893 16950 leader_election.cc:341] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 40dabdafed9140fe9787acd08ba6e484 (127.0.98.3:39447): Not found: Tablet not found: 6da69207858246f1b46effd2f41589c7
I20260619 16:53:08.227969 16950 leader_election.cc:302] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f7a800440a14ac9a5a0cf1baa974e02; no voters: 40dabdafed9140fe9787acd08ba6e484, 46b3ce200da840748de5cffca4a76fd0
I20260619 16:53:08.228062 17362 raft_consensus.cc:2726] T 6da69207858246f1b46effd2f41589c7 P 6f7a800440a14ac9a5a0cf1baa974e02 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:53:08.315294 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:08.315014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.1:52135)
0619 16:53:08.315176 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:08.315229 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":85,"threads_started":1}
I20260619 16:53:08.381197 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:08.380911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.2:46641)
0619 16:53:08.381068 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:08.381118 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":96,"threads_started":1}
I20260619 16:53:08.533273 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:08.533020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.3:40143)
0619 16:53:08.533163 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:08.533208 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
I20260619 16:53:09.316368 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:09.315982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.1:51001)
0619 16:53:09.316245 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:09.316303 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":89,"threads_started":1}
I20260619 16:53:09.382115 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:09.381847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.2:55671)
0619 16:53:09.381996 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:09.382049 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":82,"threads_started":1}
I20260619 16:53:09.534302 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:09.533906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.3:36801)
0619 16:53:09.534130 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:09.534235 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":114,"threads_started":1}
I20260619 16:53:10.317233 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:10.316973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.1:50889)
0619 16:53:10.317127 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:10.317173 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":72,"threads_started":1}
I20260619 16:53:10.382997 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:10.382688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.2:46571)
0619 16:53:10.382873 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:10.382923 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":86,"threads_started":1}
I20260619 16:53:10.535308 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:10.534927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.3:51645)
0619 16:53:10.535169 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:10.535228 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":127,"threads_started":1}
I20260619 16:53:11.318085 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:11.317781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.1:58113)
0619 16:53:11.317977 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:11.318023 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":108,"threads_started":1}
I20260619 16:53:11.383864 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:11.383571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.2:36697)
0619 16:53:11.383750 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:11.383801 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":82,"threads_started":1}
I20260619 16:53:11.536218 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:11.535918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.3:59493)
0619 16:53:11.536101 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:11.536152 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":83,"threads_started":1}
I20260619 16:53:12.319162 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:12.318817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.1:57279)
0619 16:53:12.319002 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:12.319082 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":114,"threads_started":1}
I20260619 16:53:12.384892 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:12.384585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.2:37403)
0619 16:53:12.384767 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:12.384812 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":97,"threads_started":1}
I20260619 16:53:12.537431 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:12.537096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.3:39083)
0619 16:53:12.537309 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:12.537366 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":113,"threads_started":1}
I20260619 16:53:13.320101 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:13.319808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.1:37981)
0619 16:53:13.319980 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:13.320036 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":84,"threads_started":1}
I20260619 16:53:13.385892 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:13.385586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.2:56627)
0619 16:53:13.385764 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:13.385828 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":85,"threads_started":1}
I20260619 16:53:13.538471 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:13.538101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.3:57387)
0619 16:53:13.538342 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:13.538401 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":104,"threads_started":1}
I20260619 16:53:14.321208 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:14.320899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.1:50065)
0619 16:53:14.321073 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:14.321139 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":83,"threads_started":1}
I20260619 16:53:14.386874 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:14.386606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.2:54973)
0619 16:53:14.386755 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:14.386803 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":87,"threads_started":1}
I20260619 16:53:14.529742 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 16897
I20260619 16:53:14.534967 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 17064
I20260619 16:53:14.539942 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:14.539515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41581 (local address 127.0.98.3:41537)
0619 16:53:14.539819 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:14.539881 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":159,"threads_started":1}
I20260619 16:53:14.542855 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 17198
2026-06-19T16:53:14Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58714 ms)
[----------] 30 tests from SecurityITest (252645 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:53:14 dist-test-slave-jsw9 krb5kdc[17386](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:53:14 dist-test-slave-jsw9 krb5kdc[17386](info): set up 2 sockets
Jun 19 16:53:14 dist-test-slave-jsw9 krb5kdc[17386](info): commencing operation
krb5kdc: starting...
W20260619 16:53:15.526259 392 mini_kdc.cc:121] Time spent starting KDC: real 0.962s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:53:15 dist-test-slave-jsw9 krb5kdc[17386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:53:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:53:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:53:15.660848 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46681
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46681
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:53:15.757936 17399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:15.758301 17399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:15.758369 17399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:15.759975 17399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:53:15.760033 17399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:15.760048 17399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:15.760061 17399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:53:15.760072 17399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:53:15.761884 17399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:15.762353 17399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:15.762641 17399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:15.769376 17404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:15.769505 17399 server_base.cc:1034] running on GCE node
W20260619 16:53:15.769395 17407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:15.769395 17405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:15.769943 17399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:15.770339 17399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:15.771517 17399 hybrid_clock.cc:648] HybridClock initialized: now 1781887995771505 us; error 36 us; skew 500 ppm
Jun 19 16:53:15 dist-test-slave-jsw9 krb5kdc[17386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:15.774695 17399 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:53:15.775287 17399 webserver.cc:460] Webserver started at http://127.0.98.62:34911/ using document root <none> and password file <none>
I20260619 16:53:15.775560 17399 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:15.775734 17399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:15.776013 17399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:15.777455 17399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "1fa9ae48645d496fb60fcefdfc9a13dd"
format_stamp: "Formatted at 2026-06-19 16:53:15 on dist-test-slave-jsw9"
server_key: "3937edbc3b367b0117776b14f2131671"
server_key_iv: "0d9cc0ba1ae06601a1ec3bcf867aa79e"
server_key_version: "encryptionkey@0"
I20260619 16:53:15.777846 17399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "1fa9ae48645d496fb60fcefdfc9a13dd"
format_stamp: "Formatted at 2026-06-19 16:53:15 on dist-test-slave-jsw9"
server_key: "3937edbc3b367b0117776b14f2131671"
server_key_iv: "0d9cc0ba1ae06601a1ec3bcf867aa79e"
server_key_version: "encryptionkey@0"
I20260619 16:53:15.779507 17399 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260619 16:53:15.780507 17414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:15.780684 17399 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:53:15.780777 17399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "1fa9ae48645d496fb60fcefdfc9a13dd"
format_stamp: "Formatted at 2026-06-19 16:53:15 on dist-test-slave-jsw9"
server_key: "3937edbc3b367b0117776b14f2131671"
server_key_iv: "0d9cc0ba1ae06601a1ec3bcf867aa79e"
server_key_version: "encryptionkey@0"
I20260619 16:53:15.780845 17399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:15.795221 17399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:15.798023 17399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:15.798179 17399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:15.803246 17399 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46681
I20260619 16:53:15.803326 17466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46681 every 8 connection(s)
I20260619 16:53:15.803789 17399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:53:15.805014 17467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:15.807682 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 17399
I20260619 16:53:15.807803 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:53:15.807849 17467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fa9ae48645d496fb60fcefdfc9a13dd: Bootstrap starting.
I20260619 16:53:15.808017 392 external_mini_cluster.cc:1428] Setting key 131dc796111c512b3d5d413ed8393c5b
I20260619 16:53:15.808650 17467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fa9ae48645d496fb60fcefdfc9a13dd: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:15.809038 17467 log.cc:824] T 00000000000000000000000000000000 P 1fa9ae48645d496fb60fcefdfc9a13dd: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:15.809787 17467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fa9ae48645d496fb60fcefdfc9a13dd: No bootstrap required, opened a new log
I20260619 16:53:15.811798 17467 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fa9ae48645d496fb60fcefdfc9a13dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa9ae48645d496fb60fcefdfc9a13dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46681 } }
I20260619 16:53:15.811961 17467 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fa9ae48645d496fb60fcefdfc9a13dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:15.811995 17467 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fa9ae48645d496fb60fcefdfc9a13dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fa9ae48645d496fb60fcefdfc9a13dd, State: Initialized, Role: FOLLOWER
I20260619 16:53:15.812134 17467 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fa9ae48645d496fb60fcefdfc9a13dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa9ae48645d496fb60fcefdfc9a13dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46681 } }
I20260619 16:53:15.812211 17467 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fa9ae48645d496fb60fcefdfc9a13dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:53:15.812253 17467 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fa9ae48645d496fb60fcefdfc9a13dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:53:15.812302 17467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fa9ae48645d496fb60fcefdfc9a13dd [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:15.812958 17467 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fa9ae48645d496fb60fcefdfc9a13dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa9ae48645d496fb60fcefdfc9a13dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46681 } }
I20260619 16:53:15.813064 17467 leader_election.cc:302] T 00000000000000000000000000000000 P 1fa9ae48645d496fb60fcefdfc9a13dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fa9ae48645d496fb60fcefdfc9a13dd; no voters:
I20260619 16:53:15.813242 17467 leader_election.cc:288] T 00000000000000000000000000000000 P 1fa9ae48645d496fb60fcefdfc9a13dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:53:15.813282 17472 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fa9ae48645d496fb60fcefdfc9a13dd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:53:15.813431 17472 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fa9ae48645d496fb60fcefdfc9a13dd [term 1 LEADER]: Becoming Leader. State: Replica: 1fa9ae48645d496fb60fcefdfc9a13dd, State: Running, Role: LEADER
I20260619 16:53:15.813561 17467 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fa9ae48645d496fb60fcefdfc9a13dd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:53:15.813542 17472 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fa9ae48645d496fb60fcefdfc9a13dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa9ae48645d496fb60fcefdfc9a13dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46681 } }
I20260619 16:53:15.813912 17473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fa9ae48645d496fb60fcefdfc9a13dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fa9ae48645d496fb60fcefdfc9a13dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa9ae48645d496fb60fcefdfc9a13dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46681 } } }
I20260619 16:53:15.813916 17472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fa9ae48645d496fb60fcefdfc9a13dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fa9ae48645d496fb60fcefdfc9a13dd. Latest consensus state: current_term: 1 leader_uuid: "1fa9ae48645d496fb60fcefdfc9a13dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa9ae48645d496fb60fcefdfc9a13dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46681 } } }
I20260619 16:53:15.814116 17472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fa9ae48645d496fb60fcefdfc9a13dd [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:15.815308 17473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fa9ae48645d496fb60fcefdfc9a13dd [sys.catalog]: This master's current role is: LEADER
W20260619 16:53:15.815623 17488 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1fa9ae48645d496fb60fcefdfc9a13dd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:53:15.815706 17488 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:53:15.815793 17479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:53:15.816026 17479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 16:53:15 dist-test-slave-jsw9 krb5kdc[17386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:53:15.817996 17479 catalog_manager.cc:1347] Generated new cluster ID: dc88f1e5f6b34e5295867d17457ee6d1
I20260619 16:53:15.818051 17479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:53:15.821507 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:15.808730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50342 (local address 127.0.98.62:46681)
0619 16:53:15.808961 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:15.808970 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:15.808998 (+ 28us) server_negotiation.cc:407] Connection header received
0619 16:53:15.809699 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:15.809708 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:15.809769 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:15.809929 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:15.810642 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:15.811773 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:15.812766 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:15.813271 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:15.813391 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:15.816795 (+ 3404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:15.816800 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:15.816801 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:15.816814 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:15.818889 (+ 2075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:15.819372 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:15.819375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:15.819376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:15.819491 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:15.819721 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:15.819722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:15.819722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:15.820154 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:15.820400 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:15.821062 (+ 662us) server_negotiation.cc:299] Negotiation successful
0619 16:53:15.821239 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":108,"threads_started":1}
I20260619 16:53:15.829945 17479 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:53:15.830714 17479 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:53:15.835391 17479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fa9ae48645d496fb60fcefdfc9a13dd: Generated new TSK 0
I20260619 16:53:15.835626 17479 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:53:15.866024 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46681
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:53:15.961498 17493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:15.961778 17493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:15.961812 17493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:15.963538 17493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:15.963724 17493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:15.963811 17493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:53:15.966001 17493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:15.966349 17493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:15.966637 17493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:15.972762 17498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:15.972769 17501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:15.972806 17499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:15.972846 17493 server_base.cc:1034] running on GCE node
I20260619 16:53:15.973315 17493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:15.973621 17493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:15.974787 17493 hybrid_clock.cc:648] HybridClock initialized: now 1781887995974770 us; error 39 us; skew 500 ppm
Jun 19 16:53:15 dist-test-slave-jsw9 krb5kdc[17386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:15.978042 17493 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:53:15.978678 17493 webserver.cc:460] Webserver started at http://127.0.98.1:37581/ using document root <none> and password file <none>
I20260619 16:53:15.978932 17493 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:15.979007 17493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:15.979146 17493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:15.980382 17493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-0/data/instance:
uuid: "0ae86121e2104d58bb6764b846bd5265"
format_stamp: "Formatted at 2026-06-19 16:53:15 on dist-test-slave-jsw9"
server_key: "5b544fcb086e68ff35af12e630cf2314"
server_key_iv: "7e43d911adfe0188705698db4fe4cdfc"
server_key_version: "encryptionkey@0"
I20260619 16:53:15.980762 17493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0ae86121e2104d58bb6764b846bd5265"
format_stamp: "Formatted at 2026-06-19 16:53:15 on dist-test-slave-jsw9"
server_key: "5b544fcb086e68ff35af12e630cf2314"
server_key_iv: "7e43d911adfe0188705698db4fe4cdfc"
server_key_version: "encryptionkey@0"
I20260619 16:53:15.982250 17493 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 16:53:15.983101 17508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:15.983309 17493 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:53:15.983403 17493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-0/wal
uuid: "0ae86121e2104d58bb6764b846bd5265"
format_stamp: "Formatted at 2026-06-19 16:53:15 on dist-test-slave-jsw9"
server_key: "5b544fcb086e68ff35af12e630cf2314"
server_key_iv: "7e43d911adfe0188705698db4fe4cdfc"
server_key_version: "encryptionkey@0"
I20260619 16:53:15.983507 17493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:16.004855 17493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:16.008170 17493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:16.008581 17493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:16.008965 17493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:53:16.009569 17493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:53:16.009614 17493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:16.009649 17493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:53:16.009665 17493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:16.017807 17493 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39231
I20260619 16:53:16.017866 17621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39231 every 8 connection(s)
I20260619 16:53:16.018369 17493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:53:16.023356 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 17493
I20260619 16:53:16.023494 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:53:16.023689 392 external_mini_cluster.cc:1428] Setting key 717e65e1224442d51f8538cc1ae5093e
Jun 19 16:53:16 dist-test-slave-jsw9 krb5kdc[17386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:53:16.031527 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:16.019308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47953 (local address 127.0.98.62:46681)
0619 16:53:16.019409 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:16.019412 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:16.019605 (+ 193us) server_negotiation.cc:407] Connection header received
0619 16:53:16.020872 (+ 1267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:16.020874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:16.020890 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:16.020947 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:16.021717 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:16.022330 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:16.023552 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:16.023967 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:16.024043 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:16.027512 (+ 3469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:16.027516 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:16.027516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:16.027526 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:16.029247 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:16.030000 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:16.030002 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:16.030002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:16.030086 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:16.030481 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:16.030482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:16.030482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:16.030647 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:16.030718 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:16.031327 (+ 609us) server_negotiation.cc:299] Negotiation successful
0619 16:53:16.031445 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:53:16.031713 17624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:16.019224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46681 (local address 127.0.98.1:47953)
0619 16:53:16.019459 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:16.019507 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:16.020719 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:16.021042 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:16.021046 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:16.021372 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:16.021610 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:16.021616 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:16.022417 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:16.022419 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:16.023432 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:16.023437 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:16.024082 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:16.024083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:16.024233 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:16.025048 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:53:16.025063 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:16.027298 (+ 2235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:16.029375 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:16.029390 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:16.029394 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:16.029838 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:16.030161 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:16.030162 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:16.030163 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:16.030377 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:16.030723 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:16.030731 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:16.030972 (+ 241us) client_negotiation.cc:769] Sending connection context
0619 16:53:16.031128 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 16:53:16.031383 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":99,"threads_started":1}
I20260619 16:53:16.032346 17622 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46681
I20260619 16:53:16.032476 17622 heartbeater.cc:464] Registering TS with master...
I20260619 16:53:16.032704 17622 heartbeater.cc:511] Master 127.0.98.62:46681 requested a full tablet report, sending...
I20260619 16:53:16.033525 17431 ts_manager.cc:194] Registered new tserver with Master: 0ae86121e2104d58bb6764b846bd5265 (127.0.98.1:39231)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 16:53:16.034381 17431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47953
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:53:16.059737 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46681
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:53:16.156327 17627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:16.156605 17627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:16.156635 17627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:16.158017 17627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:16.158087 17627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:16.158119 17627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:53:16.159960 17627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:16.160276 17627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:16.160569 17627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:16.166340 17635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:16.166340 17633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:16.166493 17627 server_base.cc:1034] running on GCE node
W20260619 16:53:16.166359 17632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:16.166823 17627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:16.167124 17627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:16.168267 17627 hybrid_clock.cc:648] HybridClock initialized: now 1781887996168251 us; error 36 us; skew 500 ppm
Jun 19 16:53:16 dist-test-slave-jsw9 krb5kdc[17386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:16.171010 17627 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:53:16.171509 17627 webserver.cc:460] Webserver started at http://127.0.98.2:34481/ using document root <none> and password file <none>
I20260619 16:53:16.171743 17627 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:16.171794 17627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:16.171936 17627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:16.173142 17627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-1/data/instance:
uuid: "58cf3d47195a4a739bb07eda10ec236c"
format_stamp: "Formatted at 2026-06-19 16:53:16 on dist-test-slave-jsw9"
server_key: "de8b055ee84fe76a47d127b21d529ea5"
server_key_iv: "3c2b6d5eb2e102ac7afee85a0bcd7f76"
server_key_version: "encryptionkey@0"
I20260619 16:53:16.173516 17627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-1/wal/instance:
uuid: "58cf3d47195a4a739bb07eda10ec236c"
format_stamp: "Formatted at 2026-06-19 16:53:16 on dist-test-slave-jsw9"
server_key: "de8b055ee84fe76a47d127b21d529ea5"
server_key_iv: "3c2b6d5eb2e102ac7afee85a0bcd7f76"
server_key_version: "encryptionkey@0"
I20260619 16:53:16.175017 17627 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 16:53:16.176117 17642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:16.176316 17627 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:53:16.176441 17627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-1/wal
uuid: "58cf3d47195a4a739bb07eda10ec236c"
format_stamp: "Formatted at 2026-06-19 16:53:16 on dist-test-slave-jsw9"
server_key: "de8b055ee84fe76a47d127b21d529ea5"
server_key_iv: "3c2b6d5eb2e102ac7afee85a0bcd7f76"
server_key_version: "encryptionkey@0"
I20260619 16:53:16.176519 17627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:16.193843 17627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:16.196230 17627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:16.196398 17627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:16.196656 17627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:53:16.196986 17627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:53:16.197019 17627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:16.197052 17627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:53:16.197075 17627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:16.203825 17627 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41179
I20260619 16:53:16.203867 17755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41179 every 8 connection(s)
I20260619 16:53:16.204252 17627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:53:16.206452 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 17627
I20260619 16:53:16.206570 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:53:16.206772 392 external_mini_cluster.cc:1428] Setting key f4a12f74c265cd406dfb0d983778b48f
Jun 19 16:53:16 dist-test-slave-jsw9 krb5kdc[17386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 16:53:16.216104 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:16.205080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51957 (local address 127.0.98.62:46681)
0619 16:53:16.205147 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:16.205149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:16.205363 (+ 214us) server_negotiation.cc:407] Connection header received
0619 16:53:16.206554 (+ 1191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:16.206555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:16.206570 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:16.206631 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:16.207482 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:16.208025 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:16.209437 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:16.209656 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:16.209726 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:16.212626 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:16.212628 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:16.212628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:16.212637 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:16.214273 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:16.214915 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:16.214917 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:16.214917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:16.214995 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:16.215293 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:16.215294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:16.215294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:16.215439 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:16.215503 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:16.215807 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 16:53:16.215990 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 16:53:16.216349 17758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:16.205035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46681 (local address 127.0.98.2:51957)
0619 16:53:16.205227 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:16.205269 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:16.206438 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:16.206718 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:16.206724 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:16.207094 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:16.207407 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:16.207415 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:16.208118 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:16.208119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:16.209229 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:16.209234 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:16.209748 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:16.209750 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:16.209916 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:16.210658 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:53:16.210670 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:16.212504 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:16.214361 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:16.214373 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:16.214377 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:16.214834 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:16.215061 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:16.215061 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:16.215062 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:16.215233 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:16.215499 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:16.215507 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:16.215679 (+ 172us) client_negotiation.cc:769] Sending connection context
0619 16:53:16.215782 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 16:53:16.216067 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":93,"threads_started":1}
I20260619 16:53:16.216883 17756 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46681
I20260619 16:53:16.217010 17756 heartbeater.cc:464] Registering TS with master...
I20260619 16:53:16.217237 17756 heartbeater.cc:511] Master 127.0.98.62:46681 requested a full tablet report, sending...
I20260619 16:53:16.217682 17431 ts_manager.cc:194] Registered new tserver with Master: 58cf3d47195a4a739bb07eda10ec236c (127.0.98.2:41179)
I20260619 16:53:16.218158 17431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51957
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:53:16.236714 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46681
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:53:16.341898 17761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:16.342156 17761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:16.342217 17761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:16.343502 17761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:16.343546 17761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:16.343575 17761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:53:16.345165 17761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:16.345417 17761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:16.345656 17761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:16.351136 17769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:16.351136 17767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:16.351260 17761 server_base.cc:1034] running on GCE node
W20260619 16:53:16.351423 17766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:16.351680 17761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:16.351975 17761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:16.353133 17761 hybrid_clock.cc:648] HybridClock initialized: now 1781887996353103 us; error 54 us; skew 500 ppm
Jun 19 16:53:16 dist-test-slave-jsw9 krb5kdc[17386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:16.356001 17761 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:53:16.356514 17761 webserver.cc:460] Webserver started at http://127.0.98.3:44669/ using document root <none> and password file <none>
I20260619 16:53:16.356737 17761 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:16.356779 17761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:16.356895 17761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:16.358063 17761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-2/data/instance:
uuid: "9a4f4f11c1ec4105bcf9fd2a818f8a66"
format_stamp: "Formatted at 2026-06-19 16:53:16 on dist-test-slave-jsw9"
server_key: "346e180c1d4aabcb2009532955054665"
server_key_iv: "9320542d9ec2870a14af83c8dd3b9fc8"
server_key_version: "encryptionkey@0"
I20260619 16:53:16.358449 17761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9a4f4f11c1ec4105bcf9fd2a818f8a66"
format_stamp: "Formatted at 2026-06-19 16:53:16 on dist-test-slave-jsw9"
server_key: "346e180c1d4aabcb2009532955054665"
server_key_iv: "9320542d9ec2870a14af83c8dd3b9fc8"
server_key_version: "encryptionkey@0"
I20260619 16:53:16.359793 17761 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:53:16.360615 17776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:16.360778 17761 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:53:16.360888 17761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-2/wal
uuid: "9a4f4f11c1ec4105bcf9fd2a818f8a66"
format_stamp: "Formatted at 2026-06-19 16:53:16 on dist-test-slave-jsw9"
server_key: "346e180c1d4aabcb2009532955054665"
server_key_iv: "9320542d9ec2870a14af83c8dd3b9fc8"
server_key_version: "encryptionkey@0"
I20260619 16:53:16.360958 17761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:16.377727 17761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:16.380312 17761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:16.380477 17761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:16.380739 17761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:53:16.381110 17761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:53:16.381153 17761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:16.381187 17761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:53:16.381211 17761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:16.388463 17761 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42979
I20260619 16:53:16.388525 17889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42979 every 8 connection(s)
I20260619 16:53:16.388887 17761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:53:16.392880 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 17761
I20260619 16:53:16.393002 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:53:16.393177 392 external_mini_cluster.cc:1428] Setting key 1e443226376081e10a2379037f2f6c4f
Jun 19 16:53:16 dist-test-slave-jsw9 krb5kdc[17386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:53:16.400393 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:16.389976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36325 (local address 127.0.98.62:46681)
0619 16:53:16.390066 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:16.390068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:16.390264 (+ 196us) server_negotiation.cc:407] Connection header received
0619 16:53:16.391327 (+ 1063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:16.391328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:16.391340 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:16.391406 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:16.392168 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:16.392807 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:16.394316 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:16.394541 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:16.394605 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:16.397308 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:16.397309 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:16.397310 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:16.397318 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:16.398752 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:16.399333 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:16.399335 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:16.399335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:16.399397 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:16.399686 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:16.399687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:16.399687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:16.399816 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:16.399871 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:16.400236 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 16:53:16.400312 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 16:53:16.400664 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:16.389904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46681 (local address 127.0.98.3:36325)
0619 16:53:16.390104 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:16.390154 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:16.391215 (+ 1061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:16.391505 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:16.391510 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:16.391826 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:16.392065 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:16.392070 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:16.392919 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:16.392920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:16.394250 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:16.394254 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:16.394642 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:16.394643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:16.394781 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:16.395477 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:53:16.395517 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:16.397195 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:16.398849 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:16.398858 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:16.398860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:16.399234 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:16.399496 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:16.399497 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:16.399497 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:16.399631 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:16.399906 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:16.399913 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:16.400108 (+ 195us) client_negotiation.cc:769] Sending connection context
0619 16:53:16.400220 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 16:53:16.400383 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":115,"threads_started":1}
I20260619 16:53:16.401306 17890 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46681
I20260619 16:53:16.401434 17890 heartbeater.cc:464] Registering TS with master...
I20260619 16:53:16.401650 17890 heartbeater.cc:511] Master 127.0.98.62:46681 requested a full tablet report, sending...
I20260619 16:53:16.402132 17431 ts_manager.cc:194] Registered new tserver with Master: 9a4f4f11c1ec4105bcf9fd2a818f8a66 (127.0.98.3:42979)
I20260619 16:53:16.402583 17431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36325
I20260619 16:53:16.405191 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:53:16.412876 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:16.406746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50344 (local address 127.0.98.62:46681)
0619 16:53:16.406800 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:16.406802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:16.406901 (+ 99us) server_negotiation.cc:407] Connection header received
0619 16:53:16.406959 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:16.406960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:16.406969 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:16.407010 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:16.407611 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:16.408132 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:16.408887 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:16.409087 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:16.409146 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:16.410477 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:16.410479 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:16.410479 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:16.410486 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:16.411726 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:16.412055 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:16.412056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:16.412056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:16.412102 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:16.412229 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:16.412229 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:16.412230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:16.412412 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:16.412497 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:16.412690 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 16:53:16.412768 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 16:53:16.415011 17431 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:53:16.417407 17431 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:53:16.424670 17905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:16.420278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52944 (local address 127.0.98.1:39231)
0619 16:53:16.420445 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:16.420450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:16.420533 (+ 83us) server_negotiation.cc:407] Connection header received
0619 16:53:16.420619 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:16.420622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:16.420642 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:16.420728 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:16.421036 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:16.422231 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:16.423614 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:16.424327 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:16.424385 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:16.424436 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:16.424536 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 16:53:16.424584 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
I20260619 16:53:16.424661 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:16.420016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41179 (local address 127.0.0.1:55044)
0619 16:53:16.420165 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:16.420211 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:16.420289 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:16.421135 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:16.421136 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:16.421137 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:16.421179 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:16.421180 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:16.422106 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:16.422107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:16.423531 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:16.423534 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:16.424277 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:16.424279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:16.424364 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:16.424379 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 16:53:16.424447 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 16:53:16.424517 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":13,"thread_start_us":68,"threads_started":1}
I20260619 16:53:16.424783 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:16.420173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55044 (local address 127.0.98.2:41179)
0619 16:53:16.420799 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:16.420803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:16.420814 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:53:16.420855 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:16.420857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:16.420874 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:16.420956 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:16.421242 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:16.422006 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:16.423656 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:16.424194 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:16.424272 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:16.424349 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:16.424633 (+ 284us) server_negotiation.cc:299] Negotiation successful
0619 16:53:16.424689 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":113,"threads_started":1}
I20260619 16:53:16.424867 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:16.420512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45970 (local address 127.0.98.3:42979)
0619 16:53:16.420717 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:16.420721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:16.421117 (+ 396us) server_negotiation.cc:407] Connection header received
0619 16:53:16.421169 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:16.421171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:16.421188 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:16.421272 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:16.421416 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:16.422472 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:16.423505 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:16.424232 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:16.424300 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:16.424359 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:16.424711 (+ 352us) server_negotiation.cc:299] Negotiation successful
0619 16:53:16.424758 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":86,"threads_started":1}
I20260619 16:53:16.425307 17904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:16.420031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39231 (local address 127.0.0.1:52944)
0619 16:53:16.420454 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:16.420465 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:16.420533 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:16.420870 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:16.420874 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:16.420877 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:16.420970 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:16.420973 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:16.422323 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:16.422324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:16.423531 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:16.423534 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:16.424399 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:16.424400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:16.424460 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:16.424463 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:53:16.424517 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 16:53:16.424553 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":3,"thread_start_us":108,"threads_started":1}
I20260619 16:53:16.424672 17902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:16.420243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42979 (local address 127.0.0.1:45970)
0619 16:53:16.420632 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:16.421039 (+ 407us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:16.421095 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:16.421316 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:16.421317 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:16.421319 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:16.421357 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:16.421358 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:16.422577 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:16.422578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:16.423419 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:16.423422 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:16.424394 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:16.424396 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:16.424483 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:16.424489 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 16:53:16.424535 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 16:53:16.424580 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":95,"threads_started":1}
I20260619 16:53:16.426147 17690 tablet_service.cc:1467] Processing CreateTablet for tablet c8aa7de94b4249f998a97900ea48d54a (DEFAULT_TABLE table=test-table [id=0fd07c1828f14489a911c4128ec507bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:53:16.426143 17556 tablet_service.cc:1467] Processing CreateTablet for tablet c8aa7de94b4249f998a97900ea48d54a (DEFAULT_TABLE table=test-table [id=0fd07c1828f14489a911c4128ec507bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:53:16.426358 17824 tablet_service.cc:1467] Processing CreateTablet for tablet c8aa7de94b4249f998a97900ea48d54a (DEFAULT_TABLE table=test-table [id=0fd07c1828f14489a911c4128ec507bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:53:16.426595 17690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8aa7de94b4249f998a97900ea48d54a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:16.426597 17556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8aa7de94b4249f998a97900ea48d54a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:16.426666 17824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8aa7de94b4249f998a97900ea48d54a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:16.429593 17907 tablet_bootstrap.cc:492] T c8aa7de94b4249f998a97900ea48d54a P 58cf3d47195a4a739bb07eda10ec236c: Bootstrap starting.
I20260619 16:53:16.430220 17909 tablet_bootstrap.cc:492] T c8aa7de94b4249f998a97900ea48d54a P 9a4f4f11c1ec4105bcf9fd2a818f8a66: Bootstrap starting.
I20260619 16:53:16.430495 17907 tablet_bootstrap.cc:654] T c8aa7de94b4249f998a97900ea48d54a P 58cf3d47195a4a739bb07eda10ec236c: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:16.430927 17909 tablet_bootstrap.cc:654] T c8aa7de94b4249f998a97900ea48d54a P 9a4f4f11c1ec4105bcf9fd2a818f8a66: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:16.431131 17907 log.cc:824] T c8aa7de94b4249f998a97900ea48d54a P 58cf3d47195a4a739bb07eda10ec236c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:16.431501 17909 log.cc:824] T c8aa7de94b4249f998a97900ea48d54a P 9a4f4f11c1ec4105bcf9fd2a818f8a66: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:16.431820 17908 tablet_bootstrap.cc:492] T c8aa7de94b4249f998a97900ea48d54a P 0ae86121e2104d58bb6764b846bd5265: Bootstrap starting.
I20260619 16:53:16.432016 17907 tablet_bootstrap.cc:492] T c8aa7de94b4249f998a97900ea48d54a P 58cf3d47195a4a739bb07eda10ec236c: No bootstrap required, opened a new log
I20260619 16:53:16.432094 17907 ts_tablet_manager.cc:1397] T c8aa7de94b4249f998a97900ea48d54a P 58cf3d47195a4a739bb07eda10ec236c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:53:16.432171 17909 tablet_bootstrap.cc:492] T c8aa7de94b4249f998a97900ea48d54a P 9a4f4f11c1ec4105bcf9fd2a818f8a66: No bootstrap required, opened a new log
I20260619 16:53:16.432242 17909 ts_tablet_manager.cc:1397] T c8aa7de94b4249f998a97900ea48d54a P 9a4f4f11c1ec4105bcf9fd2a818f8a66: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:53:16.432510 17908 tablet_bootstrap.cc:654] T c8aa7de94b4249f998a97900ea48d54a P 0ae86121e2104d58bb6764b846bd5265: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:16.433059 17908 log.cc:824] T c8aa7de94b4249f998a97900ea48d54a P 0ae86121e2104d58bb6764b846bd5265: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:16.433868 17908 tablet_bootstrap.cc:492] T c8aa7de94b4249f998a97900ea48d54a P 0ae86121e2104d58bb6764b846bd5265: No bootstrap required, opened a new log
I20260619 16:53:16.433941 17908 ts_tablet_manager.cc:1397] T c8aa7de94b4249f998a97900ea48d54a P 0ae86121e2104d58bb6764b846bd5265: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:53:16.434260 17909 raft_consensus.cc:348] T c8aa7de94b4249f998a97900ea48d54a P 9a4f4f11c1ec4105bcf9fd2a818f8a66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58cf3d47195a4a739bb07eda10ec236c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41179 } } peers { permanent_uuid: "0ae86121e2104d58bb6764b846bd5265" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39231 } } peers { permanent_uuid: "9a4f4f11c1ec4105bcf9fd2a818f8a66" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } }
I20260619 16:53:16.434229 17907 raft_consensus.cc:348] T c8aa7de94b4249f998a97900ea48d54a P 58cf3d47195a4a739bb07eda10ec236c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58cf3d47195a4a739bb07eda10ec236c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41179 } } peers { permanent_uuid: "0ae86121e2104d58bb6764b846bd5265" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39231 } } peers { permanent_uuid: "9a4f4f11c1ec4105bcf9fd2a818f8a66" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } }
I20260619 16:53:16.434430 17907 raft_consensus.cc:374] T c8aa7de94b4249f998a97900ea48d54a P 58cf3d47195a4a739bb07eda10ec236c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:16.434432 17909 raft_consensus.cc:374] T c8aa7de94b4249f998a97900ea48d54a P 9a4f4f11c1ec4105bcf9fd2a818f8a66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:16.434468 17907 raft_consensus.cc:729] T c8aa7de94b4249f998a97900ea48d54a P 58cf3d47195a4a739bb07eda10ec236c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58cf3d47195a4a739bb07eda10ec236c, State: Initialized, Role: FOLLOWER
I20260619 16:53:16.434482 17909 raft_consensus.cc:729] T c8aa7de94b4249f998a97900ea48d54a P 9a4f4f11c1ec4105bcf9fd2a818f8a66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4f4f11c1ec4105bcf9fd2a818f8a66, State: Initialized, Role: FOLLOWER
I20260619 16:53:16.434604 17909 consensus_queue.cc:260] T c8aa7de94b4249f998a97900ea48d54a P 9a4f4f11c1ec4105bcf9fd2a818f8a66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58cf3d47195a4a739bb07eda10ec236c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41179 } } peers { permanent_uuid: "0ae86121e2104d58bb6764b846bd5265" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39231 } } peers { permanent_uuid: "9a4f4f11c1ec4105bcf9fd2a818f8a66" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } }
I20260619 16:53:16.434712 17907 consensus_queue.cc:260] T c8aa7de94b4249f998a97900ea48d54a P 58cf3d47195a4a739bb07eda10ec236c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58cf3d47195a4a739bb07eda10ec236c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41179 } } peers { permanent_uuid: "0ae86121e2104d58bb6764b846bd5265" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39231 } } peers { permanent_uuid: "9a4f4f11c1ec4105bcf9fd2a818f8a66" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } }
I20260619 16:53:16.434957 17909 ts_tablet_manager.cc:1428] T c8aa7de94b4249f998a97900ea48d54a P 9a4f4f11c1ec4105bcf9fd2a818f8a66: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 16:53:16.434993 17890 heartbeater.cc:503] Master 127.0.98.62:46681 was elected leader, sending a full tablet report...
I20260619 16:53:16.435055 17907 ts_tablet_manager.cc:1428] T c8aa7de94b4249f998a97900ea48d54a P 58cf3d47195a4a739bb07eda10ec236c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 16:53:16.435220 17756 heartbeater.cc:503] Master 127.0.98.62:46681 was elected leader, sending a full tablet report...
I20260619 16:53:16.436659 17908 raft_consensus.cc:348] T c8aa7de94b4249f998a97900ea48d54a P 0ae86121e2104d58bb6764b846bd5265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58cf3d47195a4a739bb07eda10ec236c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41179 } } peers { permanent_uuid: "0ae86121e2104d58bb6764b846bd5265" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39231 } } peers { permanent_uuid: "9a4f4f11c1ec4105bcf9fd2a818f8a66" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } }
I20260619 16:53:16.436830 17908 raft_consensus.cc:374] T c8aa7de94b4249f998a97900ea48d54a P 0ae86121e2104d58bb6764b846bd5265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:16.436852 17908 raft_consensus.cc:729] T c8aa7de94b4249f998a97900ea48d54a P 0ae86121e2104d58bb6764b846bd5265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ae86121e2104d58bb6764b846bd5265, State: Initialized, Role: FOLLOWER
I20260619 16:53:16.436954 17908 consensus_queue.cc:260] T c8aa7de94b4249f998a97900ea48d54a P 0ae86121e2104d58bb6764b846bd5265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58cf3d47195a4a739bb07eda10ec236c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41179 } } peers { permanent_uuid: "0ae86121e2104d58bb6764b846bd5265" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39231 } } peers { permanent_uuid: "9a4f4f11c1ec4105bcf9fd2a818f8a66" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } }
I20260619 16:53:16.437232 17908 ts_tablet_manager.cc:1428] T c8aa7de94b4249f998a97900ea48d54a P 0ae86121e2104d58bb6764b846bd5265: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 16:53:16.437318 17622 heartbeater.cc:503] Master 127.0.98.62:46681 was elected leader, sending a full tablet report...
W20260619 16:53:16.454917 17757 tablet.cc:2369] T c8aa7de94b4249f998a97900ea48d54a P 58cf3d47195a4a739bb07eda10ec236c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:53:16.519225 17623 tablet.cc:2369] T c8aa7de94b4249f998a97900ea48d54a P 0ae86121e2104d58bb6764b846bd5265: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:53:16.639582 17891 tablet.cc:2369] T c8aa7de94b4249f998a97900ea48d54a P 9a4f4f11c1ec4105bcf9fd2a818f8a66: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:53:16.857191 17913 raft_consensus.cc:482] T c8aa7de94b4249f998a97900ea48d54a P 9a4f4f11c1ec4105bcf9fd2a818f8a66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:53:16.857378 17913 raft_consensus.cc:504] T c8aa7de94b4249f998a97900ea48d54a P 9a4f4f11c1ec4105bcf9fd2a818f8a66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58cf3d47195a4a739bb07eda10ec236c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41179 } } peers { permanent_uuid: "0ae86121e2104d58bb6764b846bd5265" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39231 } } peers { permanent_uuid: "9a4f4f11c1ec4105bcf9fd2a818f8a66" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } }
I20260619 16:53:16.857851 17913 leader_election.cc:288] T c8aa7de94b4249f998a97900ea48d54a P 9a4f4f11c1ec4105bcf9fd2a818f8a66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58cf3d47195a4a739bb07eda10ec236c (127.0.98.2:41179), 0ae86121e2104d58bb6764b846bd5265 (127.0.98.1:39231)
I20260619 16:53:16.860931 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:16.858222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38375 (local address 127.0.98.2:41179)
0619 16:53:16.858276 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:16.858278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:16.858400 (+ 122us) server_negotiation.cc:407] Connection header received
0619 16:53:16.858458 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:16.858459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:16.858471 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:16.858525 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:16.858671 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:16.859167 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:16.860196 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:16.860584 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:16.860640 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:16.860651 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:16.860806 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 16:53:16.860853 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 16:53:16.860970 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:16.858087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39231 (local address 127.0.98.3:46489)
0619 16:53:16.858147 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:16.858162 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:16.858245 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:16.858487 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:16.858488 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:16.858490 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:16.858580 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:16.858583 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:16.859250 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:16.859250 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:16.860104 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:16.860106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:16.860721 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:16.860722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:16.860795 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:16.860799 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:53:16.860859 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 16:53:16.860902 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":41}
I20260619 16:53:16.860970 17916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:16.858107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41179 (local address 127.0.98.3:38375)
0619 16:53:16.858323 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:16.858332 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:16.858401 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:16.858538 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:16.858538 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:16.858540 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:16.858605 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:16.858606 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:16.859249 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:16.859250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:16.860104 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:16.860106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:16.860655 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:16.860656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:16.860734 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:16.860738 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:53:16.860786 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 16:53:16.860826 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":5,"thread_start_us":140,"threads_started":1}
I20260619 16:53:16.860980 17905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:16.858257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46489 (local address 127.0.98.1:39231)
0619 16:53:16.858322 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:16.858323 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:16.858335 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:53:16.858373 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:16.858374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:16.858398 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:16.858456 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:16.858651 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:16.859168 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:16.860196 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:16.860641 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:16.860712 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:16.860725 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:16.860868 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 16:53:16.860915 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 16:53:16.861320 17576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8aa7de94b4249f998a97900ea48d54a" candidate_uuid: "9a4f4f11c1ec4105bcf9fd2a818f8a66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ae86121e2104d58bb6764b846bd5265" is_pre_election: true
I20260619 16:53:16.861330 17710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8aa7de94b4249f998a97900ea48d54a" candidate_uuid: "9a4f4f11c1ec4105bcf9fd2a818f8a66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58cf3d47195a4a739bb07eda10ec236c" is_pre_election: true
I20260619 16:53:16.861526 17576 raft_consensus.cc:2454] T c8aa7de94b4249f998a97900ea48d54a P 0ae86121e2104d58bb6764b846bd5265 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4f4f11c1ec4105bcf9fd2a818f8a66 in term 0.
I20260619 16:53:16.861526 17710 raft_consensus.cc:2454] T c8aa7de94b4249f998a97900ea48d54a P 58cf3d47195a4a739bb07eda10ec236c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a4f4f11c1ec4105bcf9fd2a818f8a66 in term 0.
I20260619 16:53:16.861796 17778 leader_election.cc:302] T c8aa7de94b4249f998a97900ea48d54a P 9a4f4f11c1ec4105bcf9fd2a818f8a66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ae86121e2104d58bb6764b846bd5265, 9a4f4f11c1ec4105bcf9fd2a818f8a66; no voters:
I20260619 16:53:16.861939 17913 raft_consensus.cc:2781] T c8aa7de94b4249f998a97900ea48d54a P 9a4f4f11c1ec4105bcf9fd2a818f8a66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:53:16.862007 17913 raft_consensus.cc:482] T c8aa7de94b4249f998a97900ea48d54a P 9a4f4f11c1ec4105bcf9fd2a818f8a66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:53:16.862030 17913 raft_consensus.cc:3037] T c8aa7de94b4249f998a97900ea48d54a P 9a4f4f11c1ec4105bcf9fd2a818f8a66 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:16.862962 17913 raft_consensus.cc:504] T c8aa7de94b4249f998a97900ea48d54a P 9a4f4f11c1ec4105bcf9fd2a818f8a66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58cf3d47195a4a739bb07eda10ec236c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41179 } } peers { permanent_uuid: "0ae86121e2104d58bb6764b846bd5265" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39231 } } peers { permanent_uuid: "9a4f4f11c1ec4105bcf9fd2a818f8a66" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } }
I20260619 16:53:16.863116 17913 leader_election.cc:288] T c8aa7de94b4249f998a97900ea48d54a P 9a4f4f11c1ec4105bcf9fd2a818f8a66 [CANDIDATE]: Term 1 election: Requested vote from peers 58cf3d47195a4a739bb07eda10ec236c (127.0.98.2:41179), 0ae86121e2104d58bb6764b846bd5265 (127.0.98.1:39231)
I20260619 16:53:16.863358 17710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8aa7de94b4249f998a97900ea48d54a" candidate_uuid: "9a4f4f11c1ec4105bcf9fd2a818f8a66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58cf3d47195a4a739bb07eda10ec236c"
I20260619 16:53:16.863376 17576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8aa7de94b4249f998a97900ea48d54a" candidate_uuid: "9a4f4f11c1ec4105bcf9fd2a818f8a66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ae86121e2104d58bb6764b846bd5265"
I20260619 16:53:16.863449 17710 raft_consensus.cc:3037] T c8aa7de94b4249f998a97900ea48d54a P 58cf3d47195a4a739bb07eda10ec236c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:16.863482 17576 raft_consensus.cc:3037] T c8aa7de94b4249f998a97900ea48d54a P 0ae86121e2104d58bb6764b846bd5265 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:16.864311 17710 raft_consensus.cc:2454] T c8aa7de94b4249f998a97900ea48d54a P 58cf3d47195a4a739bb07eda10ec236c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4f4f11c1ec4105bcf9fd2a818f8a66 in term 1.
I20260619 16:53:16.864447 17576 raft_consensus.cc:2454] T c8aa7de94b4249f998a97900ea48d54a P 0ae86121e2104d58bb6764b846bd5265 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a4f4f11c1ec4105bcf9fd2a818f8a66 in term 1.
I20260619 16:53:16.864495 17779 leader_election.cc:302] T c8aa7de94b4249f998a97900ea48d54a P 9a4f4f11c1ec4105bcf9fd2a818f8a66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58cf3d47195a4a739bb07eda10ec236c, 9a4f4f11c1ec4105bcf9fd2a818f8a66; no voters:
I20260619 16:53:16.864578 17913 raft_consensus.cc:2781] T c8aa7de94b4249f998a97900ea48d54a P 9a4f4f11c1ec4105bcf9fd2a818f8a66 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:53:16.864722 17913 raft_consensus.cc:686] T c8aa7de94b4249f998a97900ea48d54a P 9a4f4f11c1ec4105bcf9fd2a818f8a66 [term 1 LEADER]: Becoming Leader. State: Replica: 9a4f4f11c1ec4105bcf9fd2a818f8a66, State: Running, Role: LEADER
I20260619 16:53:16.864952 17913 consensus_queue.cc:237] T c8aa7de94b4249f998a97900ea48d54a P 9a4f4f11c1ec4105bcf9fd2a818f8a66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58cf3d47195a4a739bb07eda10ec236c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41179 } } peers { permanent_uuid: "0ae86121e2104d58bb6764b846bd5265" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39231 } } peers { permanent_uuid: "9a4f4f11c1ec4105bcf9fd2a818f8a66" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } }
I20260619 16:53:16.865878 17430 catalog_manager.cc:5495] T c8aa7de94b4249f998a97900ea48d54a P 9a4f4f11c1ec4105bcf9fd2a818f8a66 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a4f4f11c1ec4105bcf9fd2a818f8a66 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9a4f4f11c1ec4105bcf9fd2a818f8a66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58cf3d47195a4a739bb07eda10ec236c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ae86121e2104d58bb6764b846bd5265" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a4f4f11c1ec4105bcf9fd2a818f8a66" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } health_report { overall_health: HEALTHY } } }
I20260619 16:53:16.980178 17921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:16.977000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45976 (local address 127.0.98.3:42979)
0619 16:53:16.977230 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:16.977233 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:16.977244 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:53:16.977317 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:16.977318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:16.977345 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:16.977455 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:53:16.977768 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:16.978365 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:16.979251 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:16.979490 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:16.979556 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:16.979770 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:53:16.979933 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:53:16.980035 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:16.980051 (+ 16us) server_negotiation.cc:299] Negotiation successful
0619 16:53:16.980077 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":64,"threads_started":1}
I20260619 16:53:16.982319 17710 raft_consensus.cc:1261] T c8aa7de94b4249f998a97900ea48d54a P 58cf3d47195a4a739bb07eda10ec236c [term 1 FOLLOWER]: Refusing update from remote peer 9a4f4f11c1ec4105bcf9fd2a818f8a66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:53:16.982360 17576 raft_consensus.cc:1261] T c8aa7de94b4249f998a97900ea48d54a P 0ae86121e2104d58bb6764b846bd5265 [term 1 FOLLOWER]: Refusing update from remote peer 9a4f4f11c1ec4105bcf9fd2a818f8a66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:53:16.982724 17917 consensus_queue.cc:1035] T c8aa7de94b4249f998a97900ea48d54a P 9a4f4f11c1ec4105bcf9fd2a818f8a66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ae86121e2104d58bb6764b846bd5265" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:53:16.982893 17913 consensus_queue.cc:1035] T c8aa7de94b4249f998a97900ea48d54a P 9a4f4f11c1ec4105bcf9fd2a818f8a66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58cf3d47195a4a739bb07eda10ec236c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:53:16.985432 17922 mvcc.cc:205] Tried to move back new op lower bound from 7298613235636412416 to 7298613235159326720. Current Snapshot: MvccSnapshot[applied={T|T < 7298613235636412416}]
I20260619 16:53:16.985631 17926 mvcc.cc:205] Tried to move back new op lower bound from 7298613235636412416 to 7298613235159326720. Current Snapshot: MvccSnapshot[applied={T|T < 7298613235636412416}]
I20260619 16:53:16.986151 17925 mvcc.cc:205] Tried to move back new op lower bound from 7298613235636412416 to 7298613235159326720. Current Snapshot: MvccSnapshot[applied={T|T < 7298613235636412416}]
I20260619 16:53:16.987932 17430 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:53:16.988031 17430 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:53:16.988899 17430 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1fa9ae48645d496fb60fcefdfc9a13dd: Sending DeleteTablet for 3 replicas of tablet c8aa7de94b4249f998a97900ea48d54a
I20260619 16:53:16.989255 17690 tablet_service.cc:1514] Processing DeleteTablet for tablet c8aa7de94b4249f998a97900ea48d54a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:53:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55044
I20260619 16:53:16.989482 17824 tablet_service.cc:1514] Processing DeleteTablet for tablet c8aa7de94b4249f998a97900ea48d54a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:53:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45970
I20260619 16:53:16.989578 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 17493
I20260619 16:53:16.989702 17935 tablet_replica.cc:330] T c8aa7de94b4249f998a97900ea48d54a P 9a4f4f11c1ec4105bcf9fd2a818f8a66: stopping tablet replica
I20260619 16:53:16.989802 17935 raft_consensus.cc:2229] T c8aa7de94b4249f998a97900ea48d54a P 9a4f4f11c1ec4105bcf9fd2a818f8a66 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:53:16.989926 17935 raft_consensus.cc:2258] T c8aa7de94b4249f998a97900ea48d54a P 9a4f4f11c1ec4105bcf9fd2a818f8a66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:53:16.990334 17934 tablet_replica.cc:330] T c8aa7de94b4249f998a97900ea48d54a P 58cf3d47195a4a739bb07eda10ec236c: stopping tablet replica
I20260619 16:53:16.990455 17934 raft_consensus.cc:2229] T c8aa7de94b4249f998a97900ea48d54a P 58cf3d47195a4a739bb07eda10ec236c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:53:16.990609 17934 raft_consensus.cc:2258] T c8aa7de94b4249f998a97900ea48d54a P 58cf3d47195a4a739bb07eda10ec236c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:53:16.991150 17934 ts_tablet_manager.cc:1905] T c8aa7de94b4249f998a97900ea48d54a P 58cf3d47195a4a739bb07eda10ec236c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:53:16.992393 17935 ts_tablet_manager.cc:1905] T c8aa7de94b4249f998a97900ea48d54a P 9a4f4f11c1ec4105bcf9fd2a818f8a66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:53:16.993279 17934 ts_tablet_manager.cc:1918] T c8aa7de94b4249f998a97900ea48d54a P 58cf3d47195a4a739bb07eda10ec236c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:53:16.993446 17934 log.cc:1196] T c8aa7de94b4249f998a97900ea48d54a P 58cf3d47195a4a739bb07eda10ec236c: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-1/wal/wals/c8aa7de94b4249f998a97900ea48d54a
I20260619 16:53:16.993782 17934 ts_tablet_manager.cc:1939] T c8aa7de94b4249f998a97900ea48d54a P 58cf3d47195a4a739bb07eda10ec236c: Deleting consensus metadata
I20260619 16:53:16.994467 17417 catalog_manager.cc:4841] TS 58cf3d47195a4a739bb07eda10ec236c (127.0.98.2:41179): tablet c8aa7de94b4249f998a97900ea48d54a (table test-table [id=0fd07c1828f14489a911c4128ec507bf]) successfully deleted
I20260619 16:53:16.994354 17935 ts_tablet_manager.cc:1918] T c8aa7de94b4249f998a97900ea48d54a P 9a4f4f11c1ec4105bcf9fd2a818f8a66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:53:16.994700 17935 log.cc:1196] T c8aa7de94b4249f998a97900ea48d54a P 9a4f4f11c1ec4105bcf9fd2a818f8a66: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781887741903642-392-0/minicluster-data/ts-2/wal/wals/c8aa7de94b4249f998a97900ea48d54a
I20260619 16:53:16.995000 17935 ts_tablet_manager.cc:1939] T c8aa7de94b4249f998a97900ea48d54a P 9a4f4f11c1ec4105bcf9fd2a818f8a66: Deleting consensus metadata
I20260619 16:53:16.995524 17417 catalog_manager.cc:4841] TS 9a4f4f11c1ec4105bcf9fd2a818f8a66 (127.0.98.3:42979): tablet c8aa7de94b4249f998a97900ea48d54a (table test-table [id=0fd07c1828f14489a911c4128ec507bf]) successfully deleted
W20260619 16:53:16.996536 17416 connection.cc:535] client connection to 127.0.98.1:39231 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260619 16:53:16.996657 17416 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 16:53:16.997231 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 17627
I20260619 16:53:16.997268 17937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:16.997065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39231 (local address 127.0.0.1:52954)
0619 16:53:16.997172 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:16.997218 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":62,"threads_started":1}
W20260619 16:53:16.998052 17416 catalog_manager.cc:4584] TS 0ae86121e2104d58bb6764b846bd5265 (127.0.98.1:39231): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c8aa7de94b4249f998a97900ea48d54a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39231: connect: Connection refused (error 111)
I20260619 16:53:17.004804 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 17761
I20260619 16:53:17.009732 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 17399
2026-06-19T16:53:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2466 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T16:53:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:53:17Z Disabled control of system clock
I20260619 16:53:17.039249 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35989
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:53:17.131670 17941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:17.131963 17941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:17.131985 17941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:17.133317 17941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:53:17.133373 17941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:17.133383 17941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:17.133395 17941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:53:17.133406 17941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:53:17.134833 17941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35989
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:17.135080 17941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:17.135385 17941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:17.140477 17946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:17.140492 17947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:17.140480 17949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:17.140795 17941 server_base.cc:1034] running on GCE node
I20260619 16:53:17.141088 17941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:17.141369 17941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:17.142511 17941 hybrid_clock.cc:648] HybridClock initialized: now 1781887997142498 us; error 37 us; skew 500 ppm
I20260619 16:53:17.143849 17941 webserver.cc:460] Webserver started at http://127.0.98.62:35899/ using document root <none> and password file <none>
I20260619 16:53:17.144099 17941 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:17.144152 17941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:17.144297 17941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:17.145524 17941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "f0ecb160b5604eb5a70fa2b6f0d887e6"
format_stamp: "Formatted at 2026-06-19 16:53:17 on dist-test-slave-jsw9"
server_key: "b73f0ea50800607d565cd56c64f6085f"
server_key_iv: "ce80c65665411d731a75096791da8ff7"
server_key_version: "encryptionkey@0"
I20260619 16:53:17.145902 17941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "f0ecb160b5604eb5a70fa2b6f0d887e6"
format_stamp: "Formatted at 2026-06-19 16:53:17 on dist-test-slave-jsw9"
server_key: "b73f0ea50800607d565cd56c64f6085f"
server_key_iv: "ce80c65665411d731a75096791da8ff7"
server_key_version: "encryptionkey@0"
I20260619 16:53:17.147430 17941 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 16:53:17.148307 17955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:17.148527 17941 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:53:17.148624 17941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "f0ecb160b5604eb5a70fa2b6f0d887e6"
format_stamp: "Formatted at 2026-06-19 16:53:17 on dist-test-slave-jsw9"
server_key: "b73f0ea50800607d565cd56c64f6085f"
server_key_iv: "ce80c65665411d731a75096791da8ff7"
server_key_version: "encryptionkey@0"
I20260619 16:53:17.148696 17941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:17.182173 17941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:17.182621 17941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:17.182773 17941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:17.187394 17941 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35989
I20260619 16:53:17.187431 18007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35989 every 8 connection(s)
I20260619 16:53:17.187819 17941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:53:17.188925 18008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:17.191660 18008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0ecb160b5604eb5a70fa2b6f0d887e6: Bootstrap starting.
I20260619 16:53:17.192301 18008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0ecb160b5604eb5a70fa2b6f0d887e6: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:17.192592 18008 log.cc:824] T 00000000000000000000000000000000 P f0ecb160b5604eb5a70fa2b6f0d887e6: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:17.193507 18008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0ecb160b5604eb5a70fa2b6f0d887e6: No bootstrap required, opened a new log
I20260619 16:53:17.195181 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 17941
I20260619 16:53:17.195276 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:53:17.195494 392 external_mini_cluster.cc:1428] Setting key 9d15248f222a4a577c76ff464edc2275
I20260619 16:53:17.195468 18008 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0ecb160b5604eb5a70fa2b6f0d887e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0ecb160b5604eb5a70fa2b6f0d887e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35989 } }
I20260619 16:53:17.195650 18008 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0ecb160b5604eb5a70fa2b6f0d887e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:17.195696 18008 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0ecb160b5604eb5a70fa2b6f0d887e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0ecb160b5604eb5a70fa2b6f0d887e6, State: Initialized, Role: FOLLOWER
I20260619 16:53:17.195861 18008 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0ecb160b5604eb5a70fa2b6f0d887e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0ecb160b5604eb5a70fa2b6f0d887e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35989 } }
I20260619 16:53:17.195947 18008 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0ecb160b5604eb5a70fa2b6f0d887e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:53:17.195981 18008 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0ecb160b5604eb5a70fa2b6f0d887e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:53:17.196023 18008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0ecb160b5604eb5a70fa2b6f0d887e6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:17.196745 18008 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0ecb160b5604eb5a70fa2b6f0d887e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0ecb160b5604eb5a70fa2b6f0d887e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35989 } }
I20260619 16:53:17.197075 18008 leader_election.cc:302] T 00000000000000000000000000000000 P f0ecb160b5604eb5a70fa2b6f0d887e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0ecb160b5604eb5a70fa2b6f0d887e6; no voters:
I20260619 16:53:17.197346 18013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0ecb160b5604eb5a70fa2b6f0d887e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:53:17.197322 18008 leader_election.cc:288] T 00000000000000000000000000000000 P f0ecb160b5604eb5a70fa2b6f0d887e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:53:17.197539 18013 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0ecb160b5604eb5a70fa2b6f0d887e6 [term 1 LEADER]: Becoming Leader. State: Replica: f0ecb160b5604eb5a70fa2b6f0d887e6, State: Running, Role: LEADER
I20260619 16:53:17.197714 18008 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0ecb160b5604eb5a70fa2b6f0d887e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:53:17.197687 18013 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0ecb160b5604eb5a70fa2b6f0d887e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0ecb160b5604eb5a70fa2b6f0d887e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35989 } }
I20260619 16:53:17.198376 18015 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0ecb160b5604eb5a70fa2b6f0d887e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0ecb160b5604eb5a70fa2b6f0d887e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0ecb160b5604eb5a70fa2b6f0d887e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35989 } } }
I20260619 16:53:17.198596 18015 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0ecb160b5604eb5a70fa2b6f0d887e6 [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:17.199146 18021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:53:17.199137 18014 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0ecb160b5604eb5a70fa2b6f0d887e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0ecb160b5604eb5a70fa2b6f0d887e6. Latest consensus state: current_term: 1 leader_uuid: "f0ecb160b5604eb5a70fa2b6f0d887e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0ecb160b5604eb5a70fa2b6f0d887e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35989 } } }
I20260619 16:53:17.199565 18014 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0ecb160b5604eb5a70fa2b6f0d887e6 [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:17.199795 18021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:53:17.201779 18021 catalog_manager.cc:1347] Generated new cluster ID: d5a9db5c40ae4330a61d511651a23742
I20260619 16:53:17.201839 18021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:53:17.202350 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:17.196028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36620 (local address 127.0.98.62:35989)
0619 16:53:17.196292 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:17.196301 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:17.196330 (+ 29us) server_negotiation.cc:407] Connection header received
0619 16:53:17.196458 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:17.196464 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:17.196540 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:17.196700 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:17.196948 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:17.198706 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:17.199856 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:17.200317 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:17.200433 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:17.201250 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:17.201254 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:17.201255 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:17.201576 (+ 321us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:17.201627 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:17.201644 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:17.201719 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:17.202013 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 16:53:17.202119 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":122,"threads_started":1}
I20260619 16:53:17.210911 18021 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:53:17.211563 18021 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:53:17.217585 18021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0ecb160b5604eb5a70fa2b6f0d887e6: Generated new TSK 0
I20260619 16:53:17.217828 18021 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:53:17.223325 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35989
--builtin_ntp_servers=127.0.98.20:36721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:53:17.336639 18032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:17.336874 18032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:17.336906 18032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:17.338168 18032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:17.338238 18032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:17.338272 18032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:53:17.339710 18032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:17.339986 18032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:17.340313 18032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:17.345798 18038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:17.345820 18037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:17.346112 18032 server_base.cc:1034] running on GCE node
W20260619 16:53:17.345798 18040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:17.346441 18032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:17.346675 18032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:17.347895 18032 hybrid_clock.cc:648] HybridClock initialized: now 1781887997347872 us; error 39 us; skew 500 ppm
I20260619 16:53:17.349354 18032 webserver.cc:460] Webserver started at http://127.0.98.1:41285/ using document root <none> and password file <none>
I20260619 16:53:17.349624 18032 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:17.349677 18032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:17.349810 18032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:17.351056 18032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-0/data/instance:
uuid: "268521b49e714e18b60ca58ef9600508"
format_stamp: "Formatted at 2026-06-19 16:53:17 on dist-test-slave-jsw9"
server_key: "01e371076cbdeb6cd6964a10f9e04dd8"
server_key_iv: "5cbf75dec8697233f66c68946d4b06c1"
server_key_version: "encryptionkey@0"
I20260619 16:53:17.351433 18032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-0/wal/instance:
uuid: "268521b49e714e18b60ca58ef9600508"
format_stamp: "Formatted at 2026-06-19 16:53:17 on dist-test-slave-jsw9"
server_key: "01e371076cbdeb6cd6964a10f9e04dd8"
server_key_iv: "5cbf75dec8697233f66c68946d4b06c1"
server_key_version: "encryptionkey@0"
I20260619 16:53:17.352855 18032 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 16:53:17.353612 18046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:17.353874 18032 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:53:17.353982 18032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-0/wal
uuid: "268521b49e714e18b60ca58ef9600508"
format_stamp: "Formatted at 2026-06-19 16:53:17 on dist-test-slave-jsw9"
server_key: "01e371076cbdeb6cd6964a10f9e04dd8"
server_key_iv: "5cbf75dec8697233f66c68946d4b06c1"
server_key_version: "encryptionkey@0"
I20260619 16:53:17.354063 18032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:17.365334 18032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:17.365752 18032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:17.365901 18032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:17.366159 18032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:53:17.366609 18032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:53:17.366657 18032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:17.366691 18032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:53:17.366709 18032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:17.373307 18032 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46193
I20260619 16:53:17.373359 18159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46193 every 8 connection(s)
I20260619 16:53:17.373767 18032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:53:17.380784 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 18032
I20260619 16:53:17.380904 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:53:17.381043 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:17.374789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40853 (local address 127.0.98.62:35989)
0619 16:53:17.374904 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:17.374906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:17.375164 (+ 258us) server_negotiation.cc:407] Connection header received
0619 16:53:17.375884 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:17.375885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:17.375903 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:17.375965 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:17.376902 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:17.377818 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:17.379138 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:17.379451 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:17.379526 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:17.380699 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:17.380701 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:17.380701 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:17.380707 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:17.380729 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:17.380734 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:17.380807 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:17.380915 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 16:53:17.380959 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:53:17.381120 392 external_mini_cluster.cc:1428] Setting key 2bc95b2d4697c146fcbc603ad3ca67f2
I20260619 16:53:17.381596 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:17.374764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35989 (local address 127.0.98.1:40853)
0619 16:53:17.374978 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:17.375025 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:17.375756 (+ 731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:17.376039 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:17.376043 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:17.376054 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:17.376806 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:17.376814 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:17.377892 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:17.377893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:17.379033 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:17.379036 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:17.379559 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:17.379560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:17.379857 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:17.380527 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:53:17.380528 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:17.380564 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:53:17.380594 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:17.380810 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:17.380816 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:17.380820 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:53:17.381205 (+ 385us) client_negotiation.cc:240] Negotiation successful
0619 16:53:17.381381 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":91,"threads_started":1}
I20260619 16:53:17.382108 18160 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35989
I20260619 16:53:17.382246 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35989
--builtin_ntp_servers=127.0.98.20:36721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 16:53:17.382318 18160 heartbeater.cc:464] Registering TS with master...
I20260619 16:53:17.382582 18160 heartbeater.cc:511] Master 127.0.98.62:35989 requested a full tablet report, sending...
I20260619 16:53:17.383184 17972 ts_manager.cc:194] Registered new tserver with Master: 268521b49e714e18b60ca58ef9600508 (127.0.98.1:46193)
I20260619 16:53:17.383922 17972 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:40853
W20260619 16:53:17.477022 18163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:17.477408 18163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:17.477614 18163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:17.479143 18163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:17.479193 18163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:17.479224 18163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:53:17.480737 18163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:17.481017 18163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:17.481372 18163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:17.486756 18168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:17.486788 18163 server_base.cc:1034] running on GCE node
W20260619 16:53:17.486760 18171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:17.486756 18169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:17.487196 18163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:17.487481 18163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:17.488641 18163 hybrid_clock.cc:648] HybridClock initialized: now 1781887997488619 us; error 48 us; skew 500 ppm
I20260619 16:53:17.490032 18163 webserver.cc:460] Webserver started at http://127.0.98.2:37701/ using document root <none> and password file <none>
I20260619 16:53:17.490324 18163 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:17.490404 18163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:17.490514 18163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:17.491674 18163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-1/data/instance:
uuid: "c6c60c5392124cd59fdcdf658b07db81"
format_stamp: "Formatted at 2026-06-19 16:53:17 on dist-test-slave-jsw9"
server_key: "d25d4fc540ddd5970148be0fd7feee8e"
server_key_iv: "b9025257a6d587b830fcd9dd74a01af3"
server_key_version: "encryptionkey@0"
I20260619 16:53:17.491995 18163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c6c60c5392124cd59fdcdf658b07db81"
format_stamp: "Formatted at 2026-06-19 16:53:17 on dist-test-slave-jsw9"
server_key: "d25d4fc540ddd5970148be0fd7feee8e"
server_key_iv: "b9025257a6d587b830fcd9dd74a01af3"
server_key_version: "encryptionkey@0"
I20260619 16:53:17.493243 18163 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 16:53:17.494015 18177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:17.494230 18163 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:53:17.494319 18163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-1/wal
uuid: "c6c60c5392124cd59fdcdf658b07db81"
format_stamp: "Formatted at 2026-06-19 16:53:17 on dist-test-slave-jsw9"
server_key: "d25d4fc540ddd5970148be0fd7feee8e"
server_key_iv: "b9025257a6d587b830fcd9dd74a01af3"
server_key_version: "encryptionkey@0"
I20260619 16:53:17.494417 18163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:17.516724 18163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:17.517172 18163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:17.517349 18163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:17.517633 18163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:53:17.518039 18163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:53:17.518074 18163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:17.518107 18163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:53:17.518124 18163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:17.525996 18163 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34859
I20260619 16:53:17.526047 18290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34859 every 8 connection(s)
I20260619 16:53:17.526494 18163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:53:17.528664 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 18163
I20260619 16:53:17.528764 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:53:17.528963 392 external_mini_cluster.cc:1428] Setting key f87765ef6af7ffbd2b629425fdd4c4a4
I20260619 16:53:17.529992 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35989
--builtin_ntp_servers=127.0.98.20:36721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 16:53:17.532696 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:17.527439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53517 (local address 127.0.98.62:35989)
0619 16:53:17.527507 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:17.527509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:17.527733 (+ 224us) server_negotiation.cc:407] Connection header received
0619 16:53:17.528339 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:17.528341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:17.528360 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:17.528412 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:17.529206 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:17.529828 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:17.530942 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:17.531142 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:17.531199 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:17.532360 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:17.532361 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:17.532361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:17.532366 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:17.532387 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:17.532392 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:17.532453 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:17.532574 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 16:53:17.532613 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 16:53:17.533023 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:17.527369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35989 (local address 127.0.98.2:53517)
0619 16:53:17.527601 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:17.527641 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:17.528234 (+ 593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:17.528493 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:17.528499 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:17.528510 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:17.529118 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:17.529124 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:17.529921 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:17.529922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:17.530860 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:17.530863 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:17.531214 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:17.531215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:17.531387 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:17.532165 (+ 778us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:53:17.532165 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:17.532211 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:53:17.532246 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:17.532454 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:17.532460 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:17.532464 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:53:17.532540 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 16:53:17.532660 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":107,"threads_started":1}
I20260619 16:53:17.533628 18291 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35989
I20260619 16:53:17.533747 18291 heartbeater.cc:464] Registering TS with master...
I20260619 16:53:17.533977 18291 heartbeater.cc:511] Master 127.0.98.62:35989 requested a full tablet report, sending...
I20260619 16:53:17.534462 17972 ts_manager.cc:194] Registered new tserver with Master: c6c60c5392124cd59fdcdf658b07db81 (127.0.98.2:34859)
I20260619 16:53:17.534885 17972 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:53517
W20260619 16:53:17.621450 18294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:17.621717 18294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:17.621750 18294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:17.623050 18294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:17.623095 18294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:17.623131 18294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:53:17.624537 18294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:17.624807 18294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:17.625121 18294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:17.630231 18299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:17.630249 18302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:17.630352 18300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:17.630553 18294 server_base.cc:1034] running on GCE node
I20260619 16:53:17.630761 18294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:17.630997 18294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:17.632161 18294 hybrid_clock.cc:648] HybridClock initialized: now 1781887997632113 us; error 73 us; skew 500 ppm
I20260619 16:53:17.633806 18294 webserver.cc:460] Webserver started at http://127.0.98.3:37419/ using document root <none> and password file <none>
I20260619 16:53:17.634089 18294 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:17.634140 18294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:17.634285 18294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:17.635505 18294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-2/data/instance:
uuid: "c6db01f0a4fc4874be0e2078a24e9e59"
format_stamp: "Formatted at 2026-06-19 16:53:17 on dist-test-slave-jsw9"
server_key: "643a833d89f66906db28d5a02ed8dc13"
server_key_iv: "2aabd6b78086bb75ea26660919331a61"
server_key_version: "encryptionkey@0"
I20260619 16:53:17.635854 18294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c6db01f0a4fc4874be0e2078a24e9e59"
format_stamp: "Formatted at 2026-06-19 16:53:17 on dist-test-slave-jsw9"
server_key: "643a833d89f66906db28d5a02ed8dc13"
server_key_iv: "2aabd6b78086bb75ea26660919331a61"
server_key_version: "encryptionkey@0"
I20260619 16:53:17.637283 18294 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 16:53:17.638106 18308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:17.638293 18294 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:53:17.638398 18294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-2/wal
uuid: "c6db01f0a4fc4874be0e2078a24e9e59"
format_stamp: "Formatted at 2026-06-19 16:53:17 on dist-test-slave-jsw9"
server_key: "643a833d89f66906db28d5a02ed8dc13"
server_key_iv: "2aabd6b78086bb75ea26660919331a61"
server_key_version: "encryptionkey@0"
I20260619 16:53:17.638465 18294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:17.649145 18294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:17.649466 18294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:17.649574 18294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:17.649798 18294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:53:17.650108 18294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:53:17.650137 18294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:17.650161 18294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:53:17.650177 18294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:17.656738 18294 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39871
I20260619 16:53:17.656778 18421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39871 every 8 connection(s)
I20260619 16:53:17.657143 18294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:53:17.664911 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:17.658294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47697 (local address 127.0.98.62:35989)
0619 16:53:17.658352 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:17.658354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:17.658790 (+ 436us) server_negotiation.cc:407] Connection header received
0619 16:53:17.659613 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:17.659615 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:17.659632 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:17.659685 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:17.660719 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:17.661476 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:17.662773 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:17.662998 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:17.663130 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:17.664592 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:17.664594 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:17.664594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:17.664601 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:17.664620 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:17.664626 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:17.664674 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:17.664784 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 16:53:17.664818 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 16:53:17.665086 18424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:17.658160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35989 (local address 127.0.98.3:47697)
0619 16:53:17.658606 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:17.658657 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:17.659455 (+ 798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:17.659794 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:17.659800 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:17.659813 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:17.660590 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:17.660600 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:17.661598 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:17.661599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:17.662662 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:17.662666 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:17.663251 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:17.663254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:17.663510 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:17.664404 (+ 894us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:53:17.664405 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:17.664445 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:53:17.664479 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:17.664683 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:17.664686 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:17.664689 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:53:17.664740 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 16:53:17.664859 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":105,"threads_started":1}
I20260619 16:53:17.665781 18422 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35989
I20260619 16:53:17.665907 18422 heartbeater.cc:464] Registering TS with master...
I20260619 16:53:17.666159 18422 heartbeater.cc:511] Master 127.0.98.62:35989 requested a full tablet report, sending...
I20260619 16:53:17.666687 17972 ts_manager.cc:194] Registered new tserver with Master: c6db01f0a4fc4874be0e2078a24e9e59 (127.0.98.3:39871)
I20260619 16:53:17.666793 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 18294
I20260619 16:53:17.666906 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:53:17.667141 392 external_mini_cluster.cc:1428] Setting key 4e10a917a3dc432cf102ff8a04f2f639
I20260619 16:53:17.667238 17972 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:47697
I20260619 16:53:17.668140 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:53:17.672829 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:17.669707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36624 (local address 127.0.98.62:35989)
0619 16:53:17.669771 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:17.669773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:17.669896 (+ 123us) server_negotiation.cc:407] Connection header received
0619 16:53:17.669961 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:17.669962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:17.669974 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:17.670021 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:17.670180 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:17.670737 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:17.671524 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:17.671748 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:17.671805 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:17.672573 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:17.672575 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:17.672575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:17.672581 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:17.672599 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:17.672605 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:17.672650 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:17.672707 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 16:53:17.672737 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 16:53:17.674588 17972 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:53:17.676501 17972 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:53:17.683156 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:17.679023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39871 (local address 127.0.0.1:44072)
0619 16:53:17.679392 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:17.679444 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:17.679520 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:17.680069 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:17.680069 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:17.680072 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:17.680122 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:17.680124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:17.681041 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:17.681042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:17.682090 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:17.682093 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:17.682835 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:17.682836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:17.682916 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:17.682931 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 16:53:17.682995 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 16:53:17.683056 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":99,"threads_started":1}
I20260619 16:53:17.683333 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:17.679145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44072 (local address 127.0.98.3:39871)
0619 16:53:17.679906 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:17.679911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:17.679921 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:53:17.679964 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:17.679966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:17.679981 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:17.680060 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:17.680176 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:17.680958 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:17.682170 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:17.682753 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:17.682822 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:17.683168 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:17.683199 (+ 31us) server_negotiation.cc:299] Negotiation successful
0619 16:53:17.683244 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":165,"threads_started":1}
I20260619 16:53:17.684092 18356 tablet_service.cc:1467] Processing CreateTablet for tablet 2efc3439bead4304a43ad92a64bbf4b2 (DEFAULT_TABLE table=test-table [id=59334453cbbc4d27a82d59c3e938f053]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:53:17.684403 18435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:17.679429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35290 (local address 127.0.98.2:34859)
0619 16:53:17.679625 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:17.679628 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:17.679730 (+ 102us) server_negotiation.cc:407] Connection header received
0619 16:53:17.679798 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:17.679801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:17.679816 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:17.679892 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:17.680087 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:17.681539 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:17.682963 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:17.683904 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:17.683974 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:17.684065 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:17.684255 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 16:53:17.684305 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":118,"threads_started":1}
I20260619 16:53:17.684422 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:17.679277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34859 (local address 127.0.0.1:35290)
0619 16:53:17.679624 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:17.679633 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:17.679678 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:17.679966 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:17.679967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:17.679969 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:17.680014 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:17.680015 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:17.681654 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:17.681655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:17.682861 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:17.682864 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:17.684101 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:17.684102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:17.684162 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:17.684167 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:53:17.684231 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 16:53:17.684327 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":1,"thread_start_us":92,"threads_started":1}
I20260619 16:53:17.684513 18356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2efc3439bead4304a43ad92a64bbf4b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:17.684830 18432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:17.679167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36742 (local address 127.0.98.1:46193)
0619 16:53:17.679488 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:17.679493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:17.679505 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:53:17.679636 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:17.679639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:17.679657 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:17.679738 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:17.679996 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:17.681568 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:17.682984 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:17.684052 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:17.684109 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:17.684191 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:17.684456 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 16:53:17.684741 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":101,"threads_started":1}
I20260619 16:53:17.685264 18225 tablet_service.cc:1467] Processing CreateTablet for tablet 2efc3439bead4304a43ad92a64bbf4b2 (DEFAULT_TABLE table=test-table [id=59334453cbbc4d27a82d59c3e938f053]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:53:17.684616 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:17.679023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46193 (local address 127.0.0.1:36742)
0619 16:53:17.679428 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:17.679446 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:17.679520 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:17.679795 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:17.679798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:17.679801 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:17.679892 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:17.679895 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:17.681654 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:17.681655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:17.682904 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:17.682905 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:17.684221 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:17.684222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:17.684274 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:17.684277 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:53:17.684387 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 16:53:17.684423 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":143,"thread_start_us":98,"threads_started":1}
I20260619 16:53:17.685650 18225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2efc3439bead4304a43ad92a64bbf4b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:17.686236 18094 tablet_service.cc:1467] Processing CreateTablet for tablet 2efc3439bead4304a43ad92a64bbf4b2 (DEFAULT_TABLE table=test-table [id=59334453cbbc4d27a82d59c3e938f053]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:53:17.686558 18094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2efc3439bead4304a43ad92a64bbf4b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:17.688587 18436 tablet_bootstrap.cc:492] T 2efc3439bead4304a43ad92a64bbf4b2 P c6db01f0a4fc4874be0e2078a24e9e59: Bootstrap starting.
I20260619 16:53:17.689299 18437 tablet_bootstrap.cc:492] T 2efc3439bead4304a43ad92a64bbf4b2 P c6c60c5392124cd59fdcdf658b07db81: Bootstrap starting.
I20260619 16:53:17.689520 18436 tablet_bootstrap.cc:654] T 2efc3439bead4304a43ad92a64bbf4b2 P c6db01f0a4fc4874be0e2078a24e9e59: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:17.689585 18438 tablet_bootstrap.cc:492] T 2efc3439bead4304a43ad92a64bbf4b2 P 268521b49e714e18b60ca58ef9600508: Bootstrap starting.
I20260619 16:53:17.690058 18437 tablet_bootstrap.cc:654] T 2efc3439bead4304a43ad92a64bbf4b2 P c6c60c5392124cd59fdcdf658b07db81: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:17.690052 18436 log.cc:824] T 2efc3439bead4304a43ad92a64bbf4b2 P c6db01f0a4fc4874be0e2078a24e9e59: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:17.690279 18438 tablet_bootstrap.cc:654] T 2efc3439bead4304a43ad92a64bbf4b2 P 268521b49e714e18b60ca58ef9600508: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:17.690459 18437 log.cc:824] T 2efc3439bead4304a43ad92a64bbf4b2 P c6c60c5392124cd59fdcdf658b07db81: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:17.690630 18438 log.cc:824] T 2efc3439bead4304a43ad92a64bbf4b2 P 268521b49e714e18b60ca58ef9600508: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:17.690876 18436 tablet_bootstrap.cc:492] T 2efc3439bead4304a43ad92a64bbf4b2 P c6db01f0a4fc4874be0e2078a24e9e59: No bootstrap required, opened a new log
I20260619 16:53:17.690979 18436 ts_tablet_manager.cc:1397] T 2efc3439bead4304a43ad92a64bbf4b2 P c6db01f0a4fc4874be0e2078a24e9e59: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260619 16:53:17.691218 18437 tablet_bootstrap.cc:492] T 2efc3439bead4304a43ad92a64bbf4b2 P c6c60c5392124cd59fdcdf658b07db81: No bootstrap required, opened a new log
I20260619 16:53:17.691277 18437 ts_tablet_manager.cc:1397] T 2efc3439bead4304a43ad92a64bbf4b2 P c6c60c5392124cd59fdcdf658b07db81: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 16:53:17.693408 18437 raft_consensus.cc:348] T 2efc3439bead4304a43ad92a64bbf4b2 P c6c60c5392124cd59fdcdf658b07db81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6db01f0a4fc4874be0e2078a24e9e59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39871 } } peers { permanent_uuid: "268521b49e714e18b60ca58ef9600508" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46193 } } peers { permanent_uuid: "c6c60c5392124cd59fdcdf658b07db81" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34859 } }
I20260619 16:53:17.693444 18436 raft_consensus.cc:348] T 2efc3439bead4304a43ad92a64bbf4b2 P c6db01f0a4fc4874be0e2078a24e9e59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6db01f0a4fc4874be0e2078a24e9e59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39871 } } peers { permanent_uuid: "268521b49e714e18b60ca58ef9600508" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46193 } } peers { permanent_uuid: "c6c60c5392124cd59fdcdf658b07db81" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34859 } }
I20260619 16:53:17.693578 18436 raft_consensus.cc:374] T 2efc3439bead4304a43ad92a64bbf4b2 P c6db01f0a4fc4874be0e2078a24e9e59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:17.693583 18437 raft_consensus.cc:374] T 2efc3439bead4304a43ad92a64bbf4b2 P c6c60c5392124cd59fdcdf658b07db81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:17.693598 18436 raft_consensus.cc:729] T 2efc3439bead4304a43ad92a64bbf4b2 P c6db01f0a4fc4874be0e2078a24e9e59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6db01f0a4fc4874be0e2078a24e9e59, State: Initialized, Role: FOLLOWER
I20260619 16:53:17.693614 18437 raft_consensus.cc:729] T 2efc3439bead4304a43ad92a64bbf4b2 P c6c60c5392124cd59fdcdf658b07db81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6c60c5392124cd59fdcdf658b07db81, State: Initialized, Role: FOLLOWER
I20260619 16:53:17.693730 18436 consensus_queue.cc:260] T 2efc3439bead4304a43ad92a64bbf4b2 P c6db01f0a4fc4874be0e2078a24e9e59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6db01f0a4fc4874be0e2078a24e9e59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39871 } } peers { permanent_uuid: "268521b49e714e18b60ca58ef9600508" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46193 } } peers { permanent_uuid: "c6c60c5392124cd59fdcdf658b07db81" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34859 } }
I20260619 16:53:17.693730 18437 consensus_queue.cc:260] T 2efc3439bead4304a43ad92a64bbf4b2 P c6c60c5392124cd59fdcdf658b07db81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6db01f0a4fc4874be0e2078a24e9e59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39871 } } peers { permanent_uuid: "268521b49e714e18b60ca58ef9600508" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46193 } } peers { permanent_uuid: "c6c60c5392124cd59fdcdf658b07db81" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34859 } }
I20260619 16:53:17.694020 18436 ts_tablet_manager.cc:1428] T 2efc3439bead4304a43ad92a64bbf4b2 P c6db01f0a4fc4874be0e2078a24e9e59: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 16:53:17.694024 18437 ts_tablet_manager.cc:1428] T 2efc3439bead4304a43ad92a64bbf4b2 P c6c60c5392124cd59fdcdf658b07db81: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 16:53:17.694126 18291 heartbeater.cc:503] Master 127.0.98.62:35989 was elected leader, sending a full tablet report...
I20260619 16:53:17.694168 18422 heartbeater.cc:503] Master 127.0.98.62:35989 was elected leader, sending a full tablet report...
I20260619 16:53:17.694684 18438 tablet_bootstrap.cc:492] T 2efc3439bead4304a43ad92a64bbf4b2 P 268521b49e714e18b60ca58ef9600508: No bootstrap required, opened a new log
I20260619 16:53:17.694769 18438 ts_tablet_manager.cc:1397] T 2efc3439bead4304a43ad92a64bbf4b2 P 268521b49e714e18b60ca58ef9600508: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260619 16:53:17.697091 18438 raft_consensus.cc:348] T 2efc3439bead4304a43ad92a64bbf4b2 P 268521b49e714e18b60ca58ef9600508 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6db01f0a4fc4874be0e2078a24e9e59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39871 } } peers { permanent_uuid: "268521b49e714e18b60ca58ef9600508" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46193 } } peers { permanent_uuid: "c6c60c5392124cd59fdcdf658b07db81" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34859 } }
I20260619 16:53:17.697280 18438 raft_consensus.cc:374] T 2efc3439bead4304a43ad92a64bbf4b2 P 268521b49e714e18b60ca58ef9600508 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:17.697309 18438 raft_consensus.cc:729] T 2efc3439bead4304a43ad92a64bbf4b2 P 268521b49e714e18b60ca58ef9600508 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 268521b49e714e18b60ca58ef9600508, State: Initialized, Role: FOLLOWER
I20260619 16:53:17.697444 18438 consensus_queue.cc:260] T 2efc3439bead4304a43ad92a64bbf4b2 P 268521b49e714e18b60ca58ef9600508 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6db01f0a4fc4874be0e2078a24e9e59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39871 } } peers { permanent_uuid: "268521b49e714e18b60ca58ef9600508" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46193 } } peers { permanent_uuid: "c6c60c5392124cd59fdcdf658b07db81" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34859 } }
I20260619 16:53:17.697764 18438 ts_tablet_manager.cc:1428] T 2efc3439bead4304a43ad92a64bbf4b2 P 268521b49e714e18b60ca58ef9600508: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 16:53:17.697834 18160 heartbeater.cc:503] Master 127.0.98.62:35989 was elected leader, sending a full tablet report...
W20260619 16:53:17.777254 18292 tablet.cc:2369] T 2efc3439bead4304a43ad92a64bbf4b2 P c6c60c5392124cd59fdcdf658b07db81: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:53:17.813994 18442 raft_consensus.cc:482] T 2efc3439bead4304a43ad92a64bbf4b2 P c6c60c5392124cd59fdcdf658b07db81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:53:17.814170 18442 raft_consensus.cc:504] T 2efc3439bead4304a43ad92a64bbf4b2 P c6c60c5392124cd59fdcdf658b07db81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6db01f0a4fc4874be0e2078a24e9e59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39871 } } peers { permanent_uuid: "268521b49e714e18b60ca58ef9600508" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46193 } } peers { permanent_uuid: "c6c60c5392124cd59fdcdf658b07db81" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34859 } }
I20260619 16:53:17.814697 18442 leader_election.cc:288] T 2efc3439bead4304a43ad92a64bbf4b2 P c6c60c5392124cd59fdcdf658b07db81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6db01f0a4fc4874be0e2078a24e9e59 (127.0.98.3:39871), 268521b49e714e18b60ca58ef9600508 (127.0.98.1:46193)
I20260619 16:53:17.817991 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:17.814943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46193 (local address 127.0.98.2:49175)
0619 16:53:17.815026 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:17.815041 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:17.815096 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:17.815265 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:17.815266 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:17.815269 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:17.815356 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:17.815359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:17.816188 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:17.816189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:17.817135 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:17.817138 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:17.817705 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:17.817706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:17.817782 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:17.817787 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:53:17.817834 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 16:53:17.817894 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":84}
I20260619 16:53:17.817979 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:17.815058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57177 (local address 127.0.98.3:39871)
0619 16:53:17.815109 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:17.815111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:17.815336 (+ 225us) server_negotiation.cc:407] Connection header received
0619 16:53:17.815376 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:17.815377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:17.815389 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:17.815446 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:17.815660 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:17.816197 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:17.817244 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:17.817634 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:17.817701 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:17.817712 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:17.817852 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 16:53:17.817896 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 16:53:17.817986 18432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:17.815038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49175 (local address 127.0.98.1:46193)
0619 16:53:17.815107 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:17.815110 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:17.815122 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:53:17.815160 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:17.815161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:17.815175 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:17.815248 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:17.815484 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:17.816081 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:17.817233 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:17.817623 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:17.817683 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:17.817697 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:17.817851 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 16:53:17.817899 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 16:53:17.818005 18445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:17.814943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39871 (local address 127.0.98.2:57177)
0619 16:53:17.815276 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:17.815284 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:17.815332 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:17.815459 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:17.815460 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:17.815462 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:17.815550 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:17.815552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:17.816279 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:17.816280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:17.817147 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:17.817149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:17.817715 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:17.817716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:17.817782 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:17.817785 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:53:17.817832 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 16:53:17.817891 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":135,"thread_start_us":132,"threads_started":1}
I20260619 16:53:17.818446 18114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2efc3439bead4304a43ad92a64bbf4b2" candidate_uuid: "c6c60c5392124cd59fdcdf658b07db81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "268521b49e714e18b60ca58ef9600508" is_pre_election: true
I20260619 16:53:17.818464 18376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2efc3439bead4304a43ad92a64bbf4b2" candidate_uuid: "c6c60c5392124cd59fdcdf658b07db81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6db01f0a4fc4874be0e2078a24e9e59" is_pre_election: true
I20260619 16:53:17.818625 18376 raft_consensus.cc:2454] T 2efc3439bead4304a43ad92a64bbf4b2 P c6db01f0a4fc4874be0e2078a24e9e59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6c60c5392124cd59fdcdf658b07db81 in term 0.
I20260619 16:53:17.818625 18114 raft_consensus.cc:2454] T 2efc3439bead4304a43ad92a64bbf4b2 P 268521b49e714e18b60ca58ef9600508 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6c60c5392124cd59fdcdf658b07db81 in term 0.
I20260619 16:53:17.818868 18179 leader_election.cc:302] T 2efc3439bead4304a43ad92a64bbf4b2 P c6c60c5392124cd59fdcdf658b07db81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6c60c5392124cd59fdcdf658b07db81, c6db01f0a4fc4874be0e2078a24e9e59; no voters:
I20260619 16:53:17.819047 18442 raft_consensus.cc:2781] T 2efc3439bead4304a43ad92a64bbf4b2 P c6c60c5392124cd59fdcdf658b07db81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:53:17.819115 18442 raft_consensus.cc:482] T 2efc3439bead4304a43ad92a64bbf4b2 P c6c60c5392124cd59fdcdf658b07db81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:53:17.819140 18442 raft_consensus.cc:3037] T 2efc3439bead4304a43ad92a64bbf4b2 P c6c60c5392124cd59fdcdf658b07db81 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:17.820073 18442 raft_consensus.cc:504] T 2efc3439bead4304a43ad92a64bbf4b2 P c6c60c5392124cd59fdcdf658b07db81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6db01f0a4fc4874be0e2078a24e9e59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39871 } } peers { permanent_uuid: "268521b49e714e18b60ca58ef9600508" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46193 } } peers { permanent_uuid: "c6c60c5392124cd59fdcdf658b07db81" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34859 } }
I20260619 16:53:17.820219 18442 leader_election.cc:288] T 2efc3439bead4304a43ad92a64bbf4b2 P c6c60c5392124cd59fdcdf658b07db81 [CANDIDATE]: Term 1 election: Requested vote from peers c6db01f0a4fc4874be0e2078a24e9e59 (127.0.98.3:39871), 268521b49e714e18b60ca58ef9600508 (127.0.98.1:46193)
I20260619 16:53:17.820375 18114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2efc3439bead4304a43ad92a64bbf4b2" candidate_uuid: "c6c60c5392124cd59fdcdf658b07db81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "268521b49e714e18b60ca58ef9600508"
I20260619 16:53:17.820475 18376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2efc3439bead4304a43ad92a64bbf4b2" candidate_uuid: "c6c60c5392124cd59fdcdf658b07db81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6db01f0a4fc4874be0e2078a24e9e59"
I20260619 16:53:17.820470 18114 raft_consensus.cc:3037] T 2efc3439bead4304a43ad92a64bbf4b2 P 268521b49e714e18b60ca58ef9600508 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:17.820554 18376 raft_consensus.cc:3037] T 2efc3439bead4304a43ad92a64bbf4b2 P c6db01f0a4fc4874be0e2078a24e9e59 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:17.821528 18114 raft_consensus.cc:2454] T 2efc3439bead4304a43ad92a64bbf4b2 P 268521b49e714e18b60ca58ef9600508 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6c60c5392124cd59fdcdf658b07db81 in term 1.
I20260619 16:53:17.821529 18376 raft_consensus.cc:2454] T 2efc3439bead4304a43ad92a64bbf4b2 P c6db01f0a4fc4874be0e2078a24e9e59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6c60c5392124cd59fdcdf658b07db81 in term 1.
I20260619 16:53:17.821725 18178 leader_election.cc:302] T 2efc3439bead4304a43ad92a64bbf4b2 P c6c60c5392124cd59fdcdf658b07db81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 268521b49e714e18b60ca58ef9600508, c6c60c5392124cd59fdcdf658b07db81; no voters:
I20260619 16:53:17.821816 18442 raft_consensus.cc:2781] T 2efc3439bead4304a43ad92a64bbf4b2 P c6c60c5392124cd59fdcdf658b07db81 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:53:17.821996 18442 raft_consensus.cc:686] T 2efc3439bead4304a43ad92a64bbf4b2 P c6c60c5392124cd59fdcdf658b07db81 [term 1 LEADER]: Becoming Leader. State: Replica: c6c60c5392124cd59fdcdf658b07db81, State: Running, Role: LEADER
I20260619 16:53:17.822089 18442 consensus_queue.cc:237] T 2efc3439bead4304a43ad92a64bbf4b2 P c6c60c5392124cd59fdcdf658b07db81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6db01f0a4fc4874be0e2078a24e9e59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39871 } } peers { permanent_uuid: "268521b49e714e18b60ca58ef9600508" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46193 } } peers { permanent_uuid: "c6c60c5392124cd59fdcdf658b07db81" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34859 } }
I20260619 16:53:17.823098 17971 catalog_manager.cc:5495] T 2efc3439bead4304a43ad92a64bbf4b2 P c6c60c5392124cd59fdcdf658b07db81 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6c60c5392124cd59fdcdf658b07db81 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c6c60c5392124cd59fdcdf658b07db81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6db01f0a4fc4874be0e2078a24e9e59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "268521b49e714e18b60ca58ef9600508" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6c60c5392124cd59fdcdf658b07db81" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34859 } health_report { overall_health: HEALTHY } } }
I20260619 16:53:17.828987 18435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:17.826668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35294 (local address 127.0.98.2:34859)
0619 16:53:17.826717 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:17.826719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:17.826729 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:53:17.826767 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:17.826768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:17.826790 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:17.826889 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:53:17.826987 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:17.827463 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:17.828242 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:17.828443 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:17.828500 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:17.828610 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:53:17.828719 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:53:17.828784 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:17.828867 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 16:53:17.828884 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 16:53:17.830953 18376 raft_consensus.cc:1261] T 2efc3439bead4304a43ad92a64bbf4b2 P c6db01f0a4fc4874be0e2078a24e9e59 [term 1 FOLLOWER]: Refusing update from remote peer c6c60c5392124cd59fdcdf658b07db81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:53:17.830957 18114 raft_consensus.cc:1261] T 2efc3439bead4304a43ad92a64bbf4b2 P 268521b49e714e18b60ca58ef9600508 [term 1 FOLLOWER]: Refusing update from remote peer c6c60c5392124cd59fdcdf658b07db81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:53:17.831324 18442 consensus_queue.cc:1035] T 2efc3439bead4304a43ad92a64bbf4b2 P c6c60c5392124cd59fdcdf658b07db81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "268521b49e714e18b60ca58ef9600508" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:53:17.831467 18446 consensus_queue.cc:1035] T 2efc3439bead4304a43ad92a64bbf4b2 P c6c60c5392124cd59fdcdf658b07db81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6db01f0a4fc4874be0e2078a24e9e59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:53:17.833943 18449 mvcc.cc:205] Tried to move back new op lower bound from 7298613239112884224 to 7298613239080185856. Current Snapshot: MvccSnapshot[applied={T|T < 7298613239112884224}]
I20260619 16:53:17.834209 18450 mvcc.cc:205] Tried to move back new op lower bound from 7298613239112884224 to 7298613239080185856. Current Snapshot: MvccSnapshot[applied={T|T < 7298613239112884224}]
I20260619 16:53:17.834954 18451 mvcc.cc:205] Tried to move back new op lower bound from 7298613239112884224 to 7298613239080185856. Current Snapshot: MvccSnapshot[applied={T|T < 7298613239112884224}]
I20260619 16:53:17.837530 17971 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:53:17.837630 17971 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:53:17.838488 17971 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0ecb160b5604eb5a70fa2b6f0d887e6: Sending DeleteTablet for 3 replicas of tablet 2efc3439bead4304a43ad92a64bbf4b2
I20260619 16:53:17.838814 18356 tablet_service.cc:1514] Processing DeleteTablet for tablet 2efc3439bead4304a43ad92a64bbf4b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:53:17 UTC) from {username='slave'} at 127.0.0.1:44072
I20260619 16:53:17.838937 18225 tablet_service.cc:1514] Processing DeleteTablet for tablet 2efc3439bead4304a43ad92a64bbf4b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:53:17 UTC) from {username='slave'} at 127.0.0.1:35290
I20260619 16:53:17.838994 18094 tablet_service.cc:1514] Processing DeleteTablet for tablet 2efc3439bead4304a43ad92a64bbf4b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:53:17 UTC) from {username='slave'} at 127.0.0.1:36742
I20260619 16:53:17.839100 18458 tablet_replica.cc:330] T 2efc3439bead4304a43ad92a64bbf4b2 P c6db01f0a4fc4874be0e2078a24e9e59: stopping tablet replica
I20260619 16:53:17.839207 18458 raft_consensus.cc:2229] T 2efc3439bead4304a43ad92a64bbf4b2 P c6db01f0a4fc4874be0e2078a24e9e59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:53:17.839321 18458 raft_consensus.cc:2258] T 2efc3439bead4304a43ad92a64bbf4b2 P c6db01f0a4fc4874be0e2078a24e9e59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:53:17.839453 18459 tablet_replica.cc:330] T 2efc3439bead4304a43ad92a64bbf4b2 P c6c60c5392124cd59fdcdf658b07db81: stopping tablet replica
I20260619 16:53:17.839572 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 18032
I20260619 16:53:17.840657 18458 ts_tablet_manager.cc:1905] T 2efc3439bead4304a43ad92a64bbf4b2 P c6db01f0a4fc4874be0e2078a24e9e59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:53:17.841876 18459 raft_consensus.cc:2229] T 2efc3439bead4304a43ad92a64bbf4b2 P c6c60c5392124cd59fdcdf658b07db81 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:53:17.842031 18459 raft_consensus.cc:2258] T 2efc3439bead4304a43ad92a64bbf4b2 P c6c60c5392124cd59fdcdf658b07db81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:53:17.842514 18459 ts_tablet_manager.cc:1905] T 2efc3439bead4304a43ad92a64bbf4b2 P c6c60c5392124cd59fdcdf658b07db81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:53:17.843840 18458 ts_tablet_manager.cc:1918] T 2efc3439bead4304a43ad92a64bbf4b2 P c6db01f0a4fc4874be0e2078a24e9e59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:53:17.843868 18459 ts_tablet_manager.cc:1918] T 2efc3439bead4304a43ad92a64bbf4b2 P c6c60c5392124cd59fdcdf658b07db81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:53:17.843933 18458 log.cc:1196] T 2efc3439bead4304a43ad92a64bbf4b2 P c6db01f0a4fc4874be0e2078a24e9e59: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-2/wal/wals/2efc3439bead4304a43ad92a64bbf4b2
I20260619 16:53:17.844029 18459 log.cc:1196] T 2efc3439bead4304a43ad92a64bbf4b2 P c6c60c5392124cd59fdcdf658b07db81: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781887741903642-392-0/minicluster-data/ts-1/wal/wals/2efc3439bead4304a43ad92a64bbf4b2
I20260619 16:53:17.844218 18458 ts_tablet_manager.cc:1939] T 2efc3439bead4304a43ad92a64bbf4b2 P c6db01f0a4fc4874be0e2078a24e9e59: Deleting consensus metadata
I20260619 16:53:17.844391 18459 ts_tablet_manager.cc:1939] T 2efc3439bead4304a43ad92a64bbf4b2 P c6c60c5392124cd59fdcdf658b07db81: Deleting consensus metadata
I20260619 16:53:17.844696 17957 catalog_manager.cc:4841] TS c6db01f0a4fc4874be0e2078a24e9e59 (127.0.98.3:39871): tablet 2efc3439bead4304a43ad92a64bbf4b2 (table test-table [id=59334453cbbc4d27a82d59c3e938f053]) successfully deleted
I20260619 16:53:17.844817 17957 catalog_manager.cc:4841] TS c6c60c5392124cd59fdcdf658b07db81 (127.0.98.2:34859): tablet 2efc3439bead4304a43ad92a64bbf4b2 (table test-table [id=59334453cbbc4d27a82d59c3e938f053]) successfully deleted
W20260619 16:53:17.846535 17956 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46193 (error 108)
I20260619 16:53:17.847080 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:17.846944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46193 (local address 127.0.0.1:36744)
0619 16:53:17.846998 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:17.847046 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 16:53:17.847187 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 18163
W20260619 16:53:17.847211 17956 catalog_manager.cc:4584] TS 268521b49e714e18b60ca58ef9600508 (127.0.98.1:46193): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2efc3439bead4304a43ad92a64bbf4b2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46193: connect: Connection refused (error 111)
I20260619 16:53:17.852221 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 18294
I20260619 16:53:17.857465 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 17941
2026-06-19T16:53:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (847 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:53:17 dist-test-slave-jsw9 krb5kdc[18464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:53:17 dist-test-slave-jsw9 krb5kdc[18464](info): set up 2 sockets
Jun 19 16:53:17 dist-test-slave-jsw9 krb5kdc[18464](info): commencing operation
krb5kdc: starting...
W20260619 16:53:18.830142 392 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:53:18 dist-test-slave-jsw9 krb5kdc[18464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:53:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:53:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:53:18.962430 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38041
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38041
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:53:19.055394 18477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:19.055693 18477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:19.055730 18477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:19.057250 18477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:53:19.057305 18477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:19.057325 18477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:19.057343 18477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:53:19.057366 18477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:53:19.059157 18477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:19.059470 18477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:19.059722 18477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:19.065021 18485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:19.065022 18483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:19.065089 18477 server_base.cc:1034] running on GCE node
W20260619 16:53:19.065092 18482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:19.065541 18477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:19.065848 18477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:19.067024 18477 hybrid_clock.cc:648] HybridClock initialized: now 1781887999067005 us; error 35 us; skew 500 ppm
Jun 19 16:53:19 dist-test-slave-jsw9 krb5kdc[18464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:19.070046 18477 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:53:19.070580 18477 webserver.cc:460] Webserver started at http://127.0.98.62:33121/ using document root <none> and password file <none>
I20260619 16:53:19.070823 18477 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:19.070875 18477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:19.071009 18477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:19.072229 18477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "3b400d2bf26c45829bcee774fa37b567"
format_stamp: "Formatted at 2026-06-19 16:53:19 on dist-test-slave-jsw9"
server_key: "468e9d9cfd107e8b5bb9400084282180"
server_key_iv: "14009ef08991c4990296a4184916445c"
server_key_version: "encryptionkey@0"
I20260619 16:53:19.072590 18477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "3b400d2bf26c45829bcee774fa37b567"
format_stamp: "Formatted at 2026-06-19 16:53:19 on dist-test-slave-jsw9"
server_key: "468e9d9cfd107e8b5bb9400084282180"
server_key_iv: "14009ef08991c4990296a4184916445c"
server_key_version: "encryptionkey@0"
I20260619 16:53:19.074004 18477 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 16:53:19.074818 18492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:19.074986 18477 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:19.075065 18477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "3b400d2bf26c45829bcee774fa37b567"
format_stamp: "Formatted at 2026-06-19 16:53:19 on dist-test-slave-jsw9"
server_key: "468e9d9cfd107e8b5bb9400084282180"
server_key_iv: "14009ef08991c4990296a4184916445c"
server_key_version: "encryptionkey@0"
I20260619 16:53:19.075139 18477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:19.085508 18477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:19.087828 18477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:19.087999 18477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:19.092511 18477 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38041
I20260619 16:53:19.092549 18544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38041 every 8 connection(s)
I20260619 16:53:19.092935 18477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:53:19.093943 18545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:19.096645 18545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b400d2bf26c45829bcee774fa37b567: Bootstrap starting.
I20260619 16:53:19.097309 18545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b400d2bf26c45829bcee774fa37b567: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:19.097620 18545 log.cc:824] T 00000000000000000000000000000000 P 3b400d2bf26c45829bcee774fa37b567: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:19.098484 18545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b400d2bf26c45829bcee774fa37b567: No bootstrap required, opened a new log
I20260619 16:53:19.099352 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 18477
I20260619 16:53:19.099478 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:53:19.099670 392 external_mini_cluster.cc:1428] Setting key 6ca4b7b6d73a54a171936a2aae020baa
I20260619 16:53:19.100571 18545 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b400d2bf26c45829bcee774fa37b567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b400d2bf26c45829bcee774fa37b567" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38041 } }
I20260619 16:53:19.100768 18545 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b400d2bf26c45829bcee774fa37b567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:19.100803 18545 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b400d2bf26c45829bcee774fa37b567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b400d2bf26c45829bcee774fa37b567, State: Initialized, Role: FOLLOWER
I20260619 16:53:19.100946 18545 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b400d2bf26c45829bcee774fa37b567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b400d2bf26c45829bcee774fa37b567" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38041 } }
I20260619 16:53:19.101028 18545 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b400d2bf26c45829bcee774fa37b567 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:53:19.101060 18545 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b400d2bf26c45829bcee774fa37b567 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:53:19.101111 18545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b400d2bf26c45829bcee774fa37b567 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:19.102773 18545 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b400d2bf26c45829bcee774fa37b567 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b400d2bf26c45829bcee774fa37b567" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38041 } }
I20260619 16:53:19.102900 18545 leader_election.cc:302] T 00000000000000000000000000000000 P 3b400d2bf26c45829bcee774fa37b567 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b400d2bf26c45829bcee774fa37b567; no voters:
I20260619 16:53:19.103106 18545 leader_election.cc:288] T 00000000000000000000000000000000 P 3b400d2bf26c45829bcee774fa37b567 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:53:19.103225 18550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b400d2bf26c45829bcee774fa37b567 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:53:19.103377 18550 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b400d2bf26c45829bcee774fa37b567 [term 1 LEADER]: Becoming Leader. State: Replica: 3b400d2bf26c45829bcee774fa37b567, State: Running, Role: LEADER
I20260619 16:53:19.103530 18545 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b400d2bf26c45829bcee774fa37b567 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:53:19.103518 18550 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b400d2bf26c45829bcee774fa37b567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b400d2bf26c45829bcee774fa37b567" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38041 } }
I20260619 16:53:19.103909 18550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b400d2bf26c45829bcee774fa37b567 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b400d2bf26c45829bcee774fa37b567. Latest consensus state: current_term: 1 leader_uuid: "3b400d2bf26c45829bcee774fa37b567" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b400d2bf26c45829bcee774fa37b567" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38041 } } }
I20260619 16:53:19.103988 18550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b400d2bf26c45829bcee774fa37b567 [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:19.103957 18552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b400d2bf26c45829bcee774fa37b567 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b400d2bf26c45829bcee774fa37b567" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b400d2bf26c45829bcee774fa37b567" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38041 } } }
I20260619 16:53:19.104035 18552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b400d2bf26c45829bcee774fa37b567 [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:19.105151 18556 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:53:19.105822 18556 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 16:53:19 dist-test-slave-jsw9 krb5kdc[18464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:53:19.107590 18556 catalog_manager.cc:1347] Generated new cluster ID: 66e29612c242405c826bf562fee10b4b
I20260619 16:53:19.107641 18556 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:53:19.112488 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:19.100329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40774 (local address 127.0.98.62:38041)
0619 16:53:19.100557 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:19.100566 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:19.100600 (+ 34us) server_negotiation.cc:407] Connection header received
0619 16:53:19.100714 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:19.100720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:19.100782 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:19.100944 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:19.101911 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:19.102946 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:19.104018 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:19.104692 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:19.104782 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:19.108179 (+ 3397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:19.108185 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:19.108186 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:19.108202 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:19.110484 (+ 2282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:19.110928 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:19.110932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:19.110932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:19.111089 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:19.111308 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:19.111309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:19.111309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:19.111771 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:19.111959 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:19.112039 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 16:53:19.112229 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":127,"threads_started":1}
I20260619 16:53:19.121547 18556 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:53:19.122145 18556 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:53:19.133323 18556 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b400d2bf26c45829bcee774fa37b567: Generated new TSK 0
I20260619 16:53:19.133603 18556 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:53:19.174423 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38041
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:53:19.287662 18571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:19.287952 18571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:19.287985 18571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:19.289485 18571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:19.289544 18571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:19.289577 18571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:53:19.291545 18571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:19.291849 18571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:19.292096 18571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:19.298267 18579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:19.298267 18576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:19.298300 18577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:19.298512 18571 server_base.cc:1034] running on GCE node
I20260619 16:53:19.298766 18571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:19.299083 18571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:19.300268 18571 hybrid_clock.cc:648] HybridClock initialized: now 1781887999300254 us; error 33 us; skew 500 ppm
Jun 19 16:53:19 dist-test-slave-jsw9 krb5kdc[18464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:19.303592 18571 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:53:19.304253 18571 webserver.cc:460] Webserver started at http://127.0.98.1:38999/ using document root <none> and password file <none>
I20260619 16:53:19.304589 18571 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:19.304646 18571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:19.304800 18571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:19.306102 18571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-0/data/instance:
uuid: "86216cad23fd4146b9cd4782ed5b9acc"
format_stamp: "Formatted at 2026-06-19 16:53:19 on dist-test-slave-jsw9"
server_key: "9a8c39fe046901ba795982df32a24ed5"
server_key_iv: "f74df0af852a75a57fedaca97d6c9040"
server_key_version: "encryptionkey@0"
I20260619 16:53:19.306571 18571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-0/wal/instance:
uuid: "86216cad23fd4146b9cd4782ed5b9acc"
format_stamp: "Formatted at 2026-06-19 16:53:19 on dist-test-slave-jsw9"
server_key: "9a8c39fe046901ba795982df32a24ed5"
server_key_iv: "f74df0af852a75a57fedaca97d6c9040"
server_key_version: "encryptionkey@0"
I20260619 16:53:19.308182 18571 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 16:53:19.309162 18586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:19.309475 18571 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:53:19.309582 18571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-0/wal
uuid: "86216cad23fd4146b9cd4782ed5b9acc"
format_stamp: "Formatted at 2026-06-19 16:53:19 on dist-test-slave-jsw9"
server_key: "9a8c39fe046901ba795982df32a24ed5"
server_key_iv: "f74df0af852a75a57fedaca97d6c9040"
server_key_version: "encryptionkey@0"
I20260619 16:53:19.309675 18571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:19.323226 18571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:19.326350 18571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:19.326500 18571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:19.326761 18571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:53:19.327136 18571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:53:19.327175 18571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:19.327199 18571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:53:19.327211 18571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:19.333915 18571 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41101
I20260619 16:53:19.333976 18699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41101 every 8 connection(s)
I20260619 16:53:19.334393 18571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 16:53:19 dist-test-slave-jsw9 krb5kdc[18464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:53:19.341924 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 18571
I20260619 16:53:19.342048 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:53:19.342316 392 external_mini_cluster.cc:1428] Setting key b0a613d42e432b905373a8f5188864ff
I20260619 16:53:19.345804 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:19.335255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44557 (local address 127.0.98.62:38041)
0619 16:53:19.335332 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:19.335335 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:19.335603 (+ 268us) server_negotiation.cc:407] Connection header received
0619 16:53:19.336644 (+ 1041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:19.336645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:19.336657 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:19.336712 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:19.337369 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:19.337924 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:19.339324 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:19.339587 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:19.339665 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:19.342265 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:19.342268 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:19.342269 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:19.342283 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:19.344066 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:19.344687 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:19.344688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:19.344688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:19.344754 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:19.345045 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:19.345047 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:19.345047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:19.345222 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:19.345294 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:19.345589 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 16:53:19.345696 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 16:53:19.345916 18702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:19.335205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38041 (local address 127.0.98.1:44557)
0619 16:53:19.335465 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:19.335511 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:19.336542 (+ 1031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:19.336772 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:19.336776 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:19.337072 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:19.337296 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:19.337301 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:19.337991 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:19.337992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:19.339023 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:19.339026 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:19.339700 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:19.339701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:19.339828 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:19.340424 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:53:19.340435 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:19.342063 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:19.344185 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:19.344198 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:19.344201 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:19.344591 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:19.344822 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:19.344823 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:19.344823 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:19.344976 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:19.345289 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:19.345295 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:19.345451 (+ 156us) client_negotiation.cc:769] Sending connection context
0619 16:53:19.345553 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 16:53:19.345706 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":102,"threads_started":1}
I20260619 16:53:19.346547 18700 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38041
I20260619 16:53:19.346659 18700 heartbeater.cc:464] Registering TS with master...
I20260619 16:53:19.346859 18700 heartbeater.cc:511] Master 127.0.98.62:38041 requested a full tablet report, sending...
I20260619 16:53:19.347525 18509 ts_manager.cc:194] Registered new tserver with Master: 86216cad23fd4146b9cd4782ed5b9acc (127.0.98.1:41101)
I20260619 16:53:19.348546 18509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44557
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:53:19.371568 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38041
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:53:19.462070 18705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:19.462347 18705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:19.462388 18705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:19.463769 18705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:19.463826 18705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:19.463876 18705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:53:19.465636 18705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:19.465950 18705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:19.466219 18705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:19.471634 18713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:19.471678 18711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:19.471674 18710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:19.471907 18705 server_base.cc:1034] running on GCE node
I20260619 16:53:19.472148 18705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:19.472429 18705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:19.473579 18705 hybrid_clock.cc:648] HybridClock initialized: now 1781887999473565 us; error 32 us; skew 500 ppm
Jun 19 16:53:19 dist-test-slave-jsw9 krb5kdc[18464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:19.476442 18705 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:53:19.476953 18705 webserver.cc:460] Webserver started at http://127.0.98.2:35779/ using document root <none> and password file <none>
I20260619 16:53:19.477177 18705 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:19.477228 18705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:19.477375 18705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:19.478722 18705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-1/data/instance:
uuid: "c07ecfa64b744fc18264b61522fdeb86"
format_stamp: "Formatted at 2026-06-19 16:53:19 on dist-test-slave-jsw9"
server_key: "8e6fc86a018d690ba20bf38a07f45659"
server_key_iv: "8137eebb738e310e4e6360ee7bb68268"
server_key_version: "encryptionkey@0"
I20260619 16:53:19.479120 18705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c07ecfa64b744fc18264b61522fdeb86"
format_stamp: "Formatted at 2026-06-19 16:53:19 on dist-test-slave-jsw9"
server_key: "8e6fc86a018d690ba20bf38a07f45659"
server_key_iv: "8137eebb738e310e4e6360ee7bb68268"
server_key_version: "encryptionkey@0"
I20260619 16:53:19.480543 18705 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 16:53:19.481268 18720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:19.481442 18705 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 16:53:19.481526 18705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-1/wal
uuid: "c07ecfa64b744fc18264b61522fdeb86"
format_stamp: "Formatted at 2026-06-19 16:53:19 on dist-test-slave-jsw9"
server_key: "8e6fc86a018d690ba20bf38a07f45659"
server_key_iv: "8137eebb738e310e4e6360ee7bb68268"
server_key_version: "encryptionkey@0"
I20260619 16:53:19.481598 18705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:19.501104 18705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:19.503583 18705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:19.503739 18705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:19.504019 18705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:53:19.504405 18705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:53:19.504443 18705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:19.504477 18705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:53:19.504499 18705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:19.511610 18705 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40637
I20260619 16:53:19.511703 18833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40637 every 8 connection(s)
I20260619 16:53:19.512034 18705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:53:19.518281 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 18705
I20260619 16:53:19.518410 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:53:19.518613 392 external_mini_cluster.cc:1428] Setting key a445e2402ba743218821d9a02dde7c73
Jun 19 16:53:19 dist-test-slave-jsw9 krb5kdc[18464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:53:19.522800 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:19.512836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41221 (local address 127.0.98.62:38041)
0619 16:53:19.512899 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:19.512910 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:19.513141 (+ 231us) server_negotiation.cc:407] Connection header received
0619 16:53:19.514278 (+ 1137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:19.514281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:19.514295 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:19.514365 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:19.515183 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:19.515773 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:19.516926 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:19.517155 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:19.517229 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:19.519874 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:19.519876 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:19.519876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:19.519886 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:19.521203 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:19.521726 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:19.521727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:19.521728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:19.521798 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:19.522076 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:19.522077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:19.522077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:19.522254 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:19.522322 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:19.522619 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 16:53:19.522721 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 16:53:19.523015 18836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:19.512773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38041 (local address 127.0.98.2:41221)
0619 16:53:19.512959 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:19.513008 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:19.514074 (+ 1066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:19.514499 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:19.514508 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:19.514846 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:19.515078 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:19.515084 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:19.515885 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:19.515886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:19.516785 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:19.516787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:19.517323 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:19.517324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:19.517468 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:19.518020 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:53:19.518030 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:19.519785 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:19.521282 (+ 1497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:19.521292 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:19.521295 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:19.521634 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:19.521878 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:19.521879 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:19.521880 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:19.522013 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:19.522323 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:19.522330 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:19.522486 (+ 156us) client_negotiation.cc:769] Sending connection context
0619 16:53:19.522589 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 16:53:19.522768 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":93,"threads_started":1}
I20260619 16:53:19.523895 18834 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38041
I20260619 16:53:19.524017 18834 heartbeater.cc:464] Registering TS with master...
I20260619 16:53:19.524224 18834 heartbeater.cc:511] Master 127.0.98.62:38041 requested a full tablet report, sending...
I20260619 16:53:19.524673 18509 ts_manager.cc:194] Registered new tserver with Master: c07ecfa64b744fc18264b61522fdeb86 (127.0.98.2:40637)
I20260619 16:53:19.525090 18509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41221
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:53:19.549443 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38041
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:53:19.643105 18839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:19.643379 18839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:19.643416 18839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:19.644735 18839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:19.644796 18839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:19.644836 18839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:53:19.646683 18839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:19.646991 18839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:19.647253 18839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:19.652798 18845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:19.652935 18839 server_base.cc:1034] running on GCE node
W20260619 16:53:19.652853 18844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:19.653024 18847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:19.653295 18839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:19.653546 18839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:19.654704 18839 hybrid_clock.cc:648] HybridClock initialized: now 1781887999654686 us; error 32 us; skew 500 ppm
Jun 19 16:53:19 dist-test-slave-jsw9 krb5kdc[18464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:19.662040 18839 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:53:19.662643 18839 webserver.cc:460] Webserver started at http://127.0.98.3:39273/ using document root <none> and password file <none>
I20260619 16:53:19.662873 18839 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:19.662918 18839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:19.663038 18839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:19.664260 18839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-2/data/instance:
uuid: "ba3c05bbb12247049fa503e05799f42b"
format_stamp: "Formatted at 2026-06-19 16:53:19 on dist-test-slave-jsw9"
server_key: "0173fd465df698c3d97107bbc49646c3"
server_key_iv: "b5bd6e76df1716b13807de0069d94398"
server_key_version: "encryptionkey@0"
I20260619 16:53:19.664618 18839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ba3c05bbb12247049fa503e05799f42b"
format_stamp: "Formatted at 2026-06-19 16:53:19 on dist-test-slave-jsw9"
server_key: "0173fd465df698c3d97107bbc49646c3"
server_key_iv: "b5bd6e76df1716b13807de0069d94398"
server_key_version: "encryptionkey@0"
I20260619 16:53:19.666726 18839 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 16:53:19.668232 18854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:19.668430 18839 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:53:19.668533 18839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-2/wal
uuid: "ba3c05bbb12247049fa503e05799f42b"
format_stamp: "Formatted at 2026-06-19 16:53:19 on dist-test-slave-jsw9"
server_key: "0173fd465df698c3d97107bbc49646c3"
server_key_iv: "b5bd6e76df1716b13807de0069d94398"
server_key_version: "encryptionkey@0"
I20260619 16:53:19.668633 18839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:19.689934 18839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:19.692585 18839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:19.692749 18839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:19.693009 18839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:53:19.693356 18839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:53:19.693392 18839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:19.693424 18839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:53:19.693447 18839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:19.699926 18839 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39085
I20260619 16:53:19.699973 18967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39085 every 8 connection(s)
I20260619 16:53:19.700353 18839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:53:19.705823 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 18839
I20260619 16:53:19.705933 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:53:19.706118 392 external_mini_cluster.cc:1428] Setting key 2b59d76c77dcb2e9f35b2d91eebc6ce9
Jun 19 16:53:19 dist-test-slave-jsw9 krb5kdc[18464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:53:19.711017 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:19.701181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54565 (local address 127.0.98.62:38041)
0619 16:53:19.701239 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:19.701241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:19.701475 (+ 234us) server_negotiation.cc:407] Connection header received
0619 16:53:19.702636 (+ 1161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:19.702638 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:19.702653 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:19.702708 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:19.703468 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:19.704074 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:19.705196 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:19.705419 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:19.705478 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:19.708338 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:19.708340 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:19.708341 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:19.708350 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:19.709576 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:19.709978 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:19.709980 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:19.709980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:19.710041 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:19.710323 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:19.710324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:19.710325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:19.710461 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:19.710520 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:19.710821 (+ 301us) server_negotiation.cc:299] Negotiation successful
0619 16:53:19.710913 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 16:53:19.711212 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:19.701147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38041 (local address 127.0.98.3:54565)
0619 16:53:19.701337 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:19.701379 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:19.702514 (+ 1135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:19.702787 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:19.702791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:19.703092 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:19.703359 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:19.703365 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:19.704159 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:19.704160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:19.705126 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:19.705130 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:19.705491 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:19.705492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:19.705640 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:19.706545 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:53:19.706555 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:19.708238 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:19.709654 (+ 1416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:19.709662 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:19.709664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:19.709909 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:19.710095 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:19.710096 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:19.710096 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:19.710265 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:19.710514 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:19.710521 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:19.710681 (+ 160us) client_negotiation.cc:769] Sending connection context
0619 16:53:19.710770 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 16:53:19.710956 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":80,"threads_started":1}
I20260619 16:53:19.711700 18968 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38041
I20260619 16:53:19.711814 18968 heartbeater.cc:464] Registering TS with master...
I20260619 16:53:19.712038 18968 heartbeater.cc:511] Master 127.0.98.62:38041 requested a full tablet report, sending...
I20260619 16:53:19.712487 18509 ts_manager.cc:194] Registered new tserver with Master: ba3c05bbb12247049fa503e05799f42b (127.0.98.3:39085)
I20260619 16:53:19.712908 18509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54565
I20260619 16:53:19.718502 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:53:19.726672 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:19.720087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40776 (local address 127.0.98.62:38041)
0619 16:53:19.720134 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:19.720137 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:19.720266 (+ 129us) server_negotiation.cc:407] Connection header received
0619 16:53:19.720329 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:19.720330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:19.720341 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:19.720393 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:19.721016 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:19.721556 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:19.722348 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:19.722559 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:19.722631 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:19.724109 (+ 1478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:19.724111 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:19.724111 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:19.724190 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:19.725489 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:19.725899 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:19.725900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:19.725900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:19.725953 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:19.726108 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:19.726108 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:19.726108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:19.726312 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:19.726402 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:19.726542 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 16:53:19.726594 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 16:53:19.728668 18509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:53:19.730659 18509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:53:19.738590 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:19.734413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44274 (local address 127.0.98.1:41101)
0619 16:53:19.735022 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:19.735027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:19.735035 (+ 8us) server_negotiation.cc:407] Connection header received
0619 16:53:19.735075 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:19.735077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:19.735093 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:19.735173 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:19.735317 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:19.736343 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:19.737587 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:19.738112 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:19.738207 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:19.738281 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:19.738437 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 16:53:19.738488 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":90,"threads_started":1}
I20260619 16:53:19.738592 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:19.733996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41101 (local address 127.0.0.1:44274)
0619 16:53:19.734152 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:19.734236 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:19.734675 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:19.735183 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:19.735184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:19.735186 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:19.735238 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:19.735239 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:19.736451 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:19.736452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:19.737508 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:19.737511 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:19.738218 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:19.738220 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:19.738320 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:19.738337 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 16:53:19.738408 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 16:53:19.738474 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":99,"threads_started":1}
I20260619 16:53:19.738777 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:19.734535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34700 (local address 127.0.98.2:40637)
0619 16:53:19.734707 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:19.734712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:19.734721 (+ 9us) server_negotiation.cc:407] Connection header received
0619 16:53:19.734761 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:19.734763 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:19.734778 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:19.734850 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:19.735045 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:19.736179 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:19.737391 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:19.738115 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:19.738210 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:19.738281 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:19.738642 (+ 361us) server_negotiation.cc:299] Negotiation successful
0619 16:53:19.738693 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":89,"threads_started":1}
I20260619 16:53:19.738904 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:19.734143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39174 (local address 127.0.98.3:39085)
0619 16:53:19.734651 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:19.734657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:19.734668 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:53:19.734714 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:19.734716 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:19.734734 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:19.734821 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:19.735026 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:19.735837 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:19.737318 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:19.738115 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:19.738209 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:19.738281 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:19.738765 (+ 484us) server_negotiation.cc:299] Negotiation successful
0619 16:53:19.738820 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":134,"threads_started":1}
I20260619 16:53:19.738600 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:19.733996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39085 (local address 127.0.0.1:39174)
0619 16:53:19.734238 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:19.734252 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:19.734327 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:19.734844 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:19.734849 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:19.734852 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:19.734959 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:19.734961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:19.735949 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:19.735950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:19.737213 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:19.737217 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:19.738319 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:19.738320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:19.738389 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:19.738395 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 16:53:19.738443 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 16:53:19.738489 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":9,"thread_start_us":105,"threads_started":1}
I20260619 16:53:19.738605 18982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:19.734278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40637 (local address 127.0.0.1:34700)
0619 16:53:19.734516 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:19.734525 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:19.734575 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:19.734856 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:19.734857 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:19.734859 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:19.734960 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:19.734962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:19.736524 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:19.736525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:19.737288 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:19.737292 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:19.738318 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:19.738319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:19.738389 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:19.738394 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:53:19.738443 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 16:53:19.738503 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":16,"thread_start_us":78,"threads_started":1}
I20260619 16:53:19.739843 18902 tablet_service.cc:1467] Processing CreateTablet for tablet 965f7c212ae04e6fa5ca27b4899a9bcf (DEFAULT_TABLE table=test-table [id=738679c734a841d6be6ff658800b51e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:53:19.739840 18634 tablet_service.cc:1467] Processing CreateTablet for tablet 965f7c212ae04e6fa5ca27b4899a9bcf (DEFAULT_TABLE table=test-table [id=738679c734a841d6be6ff658800b51e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:53:19.740206 18902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 965f7c212ae04e6fa5ca27b4899a9bcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:19.740206 18634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 965f7c212ae04e6fa5ca27b4899a9bcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:19.742746 18768 tablet_service.cc:1467] Processing CreateTablet for tablet 965f7c212ae04e6fa5ca27b4899a9bcf (DEFAULT_TABLE table=test-table [id=738679c734a841d6be6ff658800b51e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:53:19.743187 18768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 965f7c212ae04e6fa5ca27b4899a9bcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:19.743602 18985 tablet_bootstrap.cc:492] T 965f7c212ae04e6fa5ca27b4899a9bcf P 86216cad23fd4146b9cd4782ed5b9acc: Bootstrap starting.
I20260619 16:53:19.743986 18986 tablet_bootstrap.cc:492] T 965f7c212ae04e6fa5ca27b4899a9bcf P ba3c05bbb12247049fa503e05799f42b: Bootstrap starting.
I20260619 16:53:19.744361 18985 tablet_bootstrap.cc:654] T 965f7c212ae04e6fa5ca27b4899a9bcf P 86216cad23fd4146b9cd4782ed5b9acc: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:19.744675 18986 tablet_bootstrap.cc:654] T 965f7c212ae04e6fa5ca27b4899a9bcf P ba3c05bbb12247049fa503e05799f42b: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:19.744902 18985 log.cc:824] T 965f7c212ae04e6fa5ca27b4899a9bcf P 86216cad23fd4146b9cd4782ed5b9acc: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:19.745246 18986 log.cc:824] T 965f7c212ae04e6fa5ca27b4899a9bcf P ba3c05bbb12247049fa503e05799f42b: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:19.745867 18985 tablet_bootstrap.cc:492] T 965f7c212ae04e6fa5ca27b4899a9bcf P 86216cad23fd4146b9cd4782ed5b9acc: No bootstrap required, opened a new log
I20260619 16:53:19.745929 18985 ts_tablet_manager.cc:1397] T 965f7c212ae04e6fa5ca27b4899a9bcf P 86216cad23fd4146b9cd4782ed5b9acc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:53:19.746038 18986 tablet_bootstrap.cc:492] T 965f7c212ae04e6fa5ca27b4899a9bcf P ba3c05bbb12247049fa503e05799f42b: No bootstrap required, opened a new log
I20260619 16:53:19.746110 18986 ts_tablet_manager.cc:1397] T 965f7c212ae04e6fa5ca27b4899a9bcf P ba3c05bbb12247049fa503e05799f42b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:53:19.747164 18989 tablet_bootstrap.cc:492] T 965f7c212ae04e6fa5ca27b4899a9bcf P c07ecfa64b744fc18264b61522fdeb86: Bootstrap starting.
I20260619 16:53:19.748260 18989 tablet_bootstrap.cc:654] T 965f7c212ae04e6fa5ca27b4899a9bcf P c07ecfa64b744fc18264b61522fdeb86: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:19.748152 18985 raft_consensus.cc:348] T 965f7c212ae04e6fa5ca27b4899a9bcf P 86216cad23fd4146b9cd4782ed5b9acc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86216cad23fd4146b9cd4782ed5b9acc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "ba3c05bbb12247049fa503e05799f42b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39085 } } peers { permanent_uuid: "c07ecfa64b744fc18264b61522fdeb86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40637 } }
I20260619 16:53:19.748374 18985 raft_consensus.cc:374] T 965f7c212ae04e6fa5ca27b4899a9bcf P 86216cad23fd4146b9cd4782ed5b9acc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:19.748430 18985 raft_consensus.cc:729] T 965f7c212ae04e6fa5ca27b4899a9bcf P 86216cad23fd4146b9cd4782ed5b9acc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86216cad23fd4146b9cd4782ed5b9acc, State: Initialized, Role: FOLLOWER
I20260619 16:53:19.748400 18986 raft_consensus.cc:348] T 965f7c212ae04e6fa5ca27b4899a9bcf P ba3c05bbb12247049fa503e05799f42b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86216cad23fd4146b9cd4782ed5b9acc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "ba3c05bbb12247049fa503e05799f42b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39085 } } peers { permanent_uuid: "c07ecfa64b744fc18264b61522fdeb86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40637 } }
I20260619 16:53:19.748585 18986 raft_consensus.cc:374] T 965f7c212ae04e6fa5ca27b4899a9bcf P ba3c05bbb12247049fa503e05799f42b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:19.748625 18986 raft_consensus.cc:729] T 965f7c212ae04e6fa5ca27b4899a9bcf P ba3c05bbb12247049fa503e05799f42b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba3c05bbb12247049fa503e05799f42b, State: Initialized, Role: FOLLOWER
I20260619 16:53:19.748582 18985 consensus_queue.cc:260] T 965f7c212ae04e6fa5ca27b4899a9bcf P 86216cad23fd4146b9cd4782ed5b9acc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86216cad23fd4146b9cd4782ed5b9acc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "ba3c05bbb12247049fa503e05799f42b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39085 } } peers { permanent_uuid: "c07ecfa64b744fc18264b61522fdeb86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40637 } }
I20260619 16:53:19.748765 18986 consensus_queue.cc:260] T 965f7c212ae04e6fa5ca27b4899a9bcf P ba3c05bbb12247049fa503e05799f42b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86216cad23fd4146b9cd4782ed5b9acc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "ba3c05bbb12247049fa503e05799f42b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39085 } } peers { permanent_uuid: "c07ecfa64b744fc18264b61522fdeb86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40637 } }
I20260619 16:53:19.748975 18985 ts_tablet_manager.cc:1428] T 965f7c212ae04e6fa5ca27b4899a9bcf P 86216cad23fd4146b9cd4782ed5b9acc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 16:53:19.748975 18989 log.cc:824] T 965f7c212ae04e6fa5ca27b4899a9bcf P c07ecfa64b744fc18264b61522fdeb86: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:19.749042 18700 heartbeater.cc:503] Master 127.0.98.62:38041 was elected leader, sending a full tablet report...
I20260619 16:53:19.749071 18986 ts_tablet_manager.cc:1428] T 965f7c212ae04e6fa5ca27b4899a9bcf P ba3c05bbb12247049fa503e05799f42b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 16:53:19.749172 18968 heartbeater.cc:503] Master 127.0.98.62:38041 was elected leader, sending a full tablet report...
I20260619 16:53:19.750489 18989 tablet_bootstrap.cc:492] T 965f7c212ae04e6fa5ca27b4899a9bcf P c07ecfa64b744fc18264b61522fdeb86: No bootstrap required, opened a new log
I20260619 16:53:19.750568 18989 ts_tablet_manager.cc:1397] T 965f7c212ae04e6fa5ca27b4899a9bcf P c07ecfa64b744fc18264b61522fdeb86: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:53:19.752125 18989 raft_consensus.cc:348] T 965f7c212ae04e6fa5ca27b4899a9bcf P c07ecfa64b744fc18264b61522fdeb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86216cad23fd4146b9cd4782ed5b9acc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "ba3c05bbb12247049fa503e05799f42b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39085 } } peers { permanent_uuid: "c07ecfa64b744fc18264b61522fdeb86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40637 } }
I20260619 16:53:19.752279 18989 raft_consensus.cc:374] T 965f7c212ae04e6fa5ca27b4899a9bcf P c07ecfa64b744fc18264b61522fdeb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:19.752303 18989 raft_consensus.cc:729] T 965f7c212ae04e6fa5ca27b4899a9bcf P c07ecfa64b744fc18264b61522fdeb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c07ecfa64b744fc18264b61522fdeb86, State: Initialized, Role: FOLLOWER
I20260619 16:53:19.752413 18989 consensus_queue.cc:260] T 965f7c212ae04e6fa5ca27b4899a9bcf P c07ecfa64b744fc18264b61522fdeb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86216cad23fd4146b9cd4782ed5b9acc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "ba3c05bbb12247049fa503e05799f42b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39085 } } peers { permanent_uuid: "c07ecfa64b744fc18264b61522fdeb86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40637 } }
I20260619 16:53:19.752710 18989 ts_tablet_manager.cc:1428] T 965f7c212ae04e6fa5ca27b4899a9bcf P c07ecfa64b744fc18264b61522fdeb86: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:53:19.752846 18834 heartbeater.cc:503] Master 127.0.98.62:38041 was elected leader, sending a full tablet report...
W20260619 16:53:19.762621 18835 tablet.cc:2369] T 965f7c212ae04e6fa5ca27b4899a9bcf P c07ecfa64b744fc18264b61522fdeb86: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:53:19.835237 18701 tablet.cc:2369] T 965f7c212ae04e6fa5ca27b4899a9bcf P 86216cad23fd4146b9cd4782ed5b9acc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:53:19.951030 18969 tablet.cc:2369] T 965f7c212ae04e6fa5ca27b4899a9bcf P ba3c05bbb12247049fa503e05799f42b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:53:19.954852 18991 raft_consensus.cc:482] T 965f7c212ae04e6fa5ca27b4899a9bcf P ba3c05bbb12247049fa503e05799f42b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:53:19.955001 18991 raft_consensus.cc:504] T 965f7c212ae04e6fa5ca27b4899a9bcf P ba3c05bbb12247049fa503e05799f42b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86216cad23fd4146b9cd4782ed5b9acc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "ba3c05bbb12247049fa503e05799f42b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39085 } } peers { permanent_uuid: "c07ecfa64b744fc18264b61522fdeb86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40637 } }
I20260619 16:53:19.955470 18991 leader_election.cc:288] T 965f7c212ae04e6fa5ca27b4899a9bcf P ba3c05bbb12247049fa503e05799f42b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86216cad23fd4146b9cd4782ed5b9acc (127.0.98.1:41101), c07ecfa64b744fc18264b61522fdeb86 (127.0.98.2:40637)
I20260619 16:53:19.958606 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:19.955833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35507 (local address 127.0.98.2:40637)
0619 16:53:19.955884 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:19.955887 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:19.956084 (+ 197us) server_negotiation.cc:407] Connection header received
0619 16:53:19.956124 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:19.956126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:19.956140 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:19.956193 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:19.956393 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:19.956882 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:19.957829 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:19.958263 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:19.958332 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:19.958343 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:19.958492 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 16:53:19.958538 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 16:53:19.958591 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:19.955800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46207 (local address 127.0.98.1:41101)
0619 16:53:19.955857 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:19.955859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:19.955874 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:53:19.955936 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:19.955937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:19.955950 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:19.956037 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:19.956211 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:19.956795 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:19.957815 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:19.958245 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:19.958307 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:19.958329 (+ 22us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:19.958469 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 16:53:19.958513 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 16:53:19.958592 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:19.955726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41101 (local address 127.0.98.3:46207)
0619 16:53:19.955799 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:19.955813 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:19.955861 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:19.956038 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:19.956039 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:19.956040 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:19.956137 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:19.956140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:19.956889 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:19.956891 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:19.957721 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:19.957723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:19.958318 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:19.958319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:19.958401 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:19.958405 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:53:19.958450 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 16:53:19.958499 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 16:53:19.958607 18994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:19.955726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40637 (local address 127.0.98.3:35507)
0619 16:53:19.956001 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:19.956011 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:19.956077 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:19.956209 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:19.956209 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:19.956211 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:19.956285 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:19.956286 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:19.956970 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:19.956971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:19.957745 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:19.957748 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:19.958348 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:19.958349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:19.958425 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:19.958428 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:53:19.958480 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 16:53:19.958518 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":83,"thread_start_us":120,"threads_started":1}
I20260619 16:53:19.959025 18654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965f7c212ae04e6fa5ca27b4899a9bcf" candidate_uuid: "ba3c05bbb12247049fa503e05799f42b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86216cad23fd4146b9cd4782ed5b9acc" is_pre_election: true
I20260619 16:53:19.959239 18654 raft_consensus.cc:2454] T 965f7c212ae04e6fa5ca27b4899a9bcf P 86216cad23fd4146b9cd4782ed5b9acc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba3c05bbb12247049fa503e05799f42b in term 0.
I20260619 16:53:19.959257 18788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965f7c212ae04e6fa5ca27b4899a9bcf" candidate_uuid: "ba3c05bbb12247049fa503e05799f42b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c07ecfa64b744fc18264b61522fdeb86" is_pre_election: true
I20260619 16:53:19.959420 18788 raft_consensus.cc:2454] T 965f7c212ae04e6fa5ca27b4899a9bcf P c07ecfa64b744fc18264b61522fdeb86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba3c05bbb12247049fa503e05799f42b in term 0.
I20260619 16:53:19.959472 18857 leader_election.cc:302] T 965f7c212ae04e6fa5ca27b4899a9bcf P ba3c05bbb12247049fa503e05799f42b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86216cad23fd4146b9cd4782ed5b9acc, ba3c05bbb12247049fa503e05799f42b; no voters:
I20260619 16:53:19.959671 18991 raft_consensus.cc:2781] T 965f7c212ae04e6fa5ca27b4899a9bcf P ba3c05bbb12247049fa503e05799f42b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:53:19.959719 18991 raft_consensus.cc:482] T 965f7c212ae04e6fa5ca27b4899a9bcf P ba3c05bbb12247049fa503e05799f42b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:53:19.959734 18991 raft_consensus.cc:3037] T 965f7c212ae04e6fa5ca27b4899a9bcf P ba3c05bbb12247049fa503e05799f42b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:19.960544 18991 raft_consensus.cc:504] T 965f7c212ae04e6fa5ca27b4899a9bcf P ba3c05bbb12247049fa503e05799f42b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86216cad23fd4146b9cd4782ed5b9acc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "ba3c05bbb12247049fa503e05799f42b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39085 } } peers { permanent_uuid: "c07ecfa64b744fc18264b61522fdeb86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40637 } }
I20260619 16:53:19.960685 18991 leader_election.cc:288] T 965f7c212ae04e6fa5ca27b4899a9bcf P ba3c05bbb12247049fa503e05799f42b [CANDIDATE]: Term 1 election: Requested vote from peers 86216cad23fd4146b9cd4782ed5b9acc (127.0.98.1:41101), c07ecfa64b744fc18264b61522fdeb86 (127.0.98.2:40637)
I20260619 16:53:19.960879 18788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965f7c212ae04e6fa5ca27b4899a9bcf" candidate_uuid: "ba3c05bbb12247049fa503e05799f42b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c07ecfa64b744fc18264b61522fdeb86"
I20260619 16:53:19.960968 18788 raft_consensus.cc:3037] T 965f7c212ae04e6fa5ca27b4899a9bcf P c07ecfa64b744fc18264b61522fdeb86 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:19.960950 18654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965f7c212ae04e6fa5ca27b4899a9bcf" candidate_uuid: "ba3c05bbb12247049fa503e05799f42b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86216cad23fd4146b9cd4782ed5b9acc"
I20260619 16:53:19.961066 18654 raft_consensus.cc:3037] T 965f7c212ae04e6fa5ca27b4899a9bcf P 86216cad23fd4146b9cd4782ed5b9acc [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:19.961930 18788 raft_consensus.cc:2454] T 965f7c212ae04e6fa5ca27b4899a9bcf P c07ecfa64b744fc18264b61522fdeb86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba3c05bbb12247049fa503e05799f42b in term 1.
I20260619 16:53:19.961930 18654 raft_consensus.cc:2454] T 965f7c212ae04e6fa5ca27b4899a9bcf P 86216cad23fd4146b9cd4782ed5b9acc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba3c05bbb12247049fa503e05799f42b in term 1.
I20260619 16:53:19.962117 18858 leader_election.cc:302] T 965f7c212ae04e6fa5ca27b4899a9bcf P ba3c05bbb12247049fa503e05799f42b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba3c05bbb12247049fa503e05799f42b, c07ecfa64b744fc18264b61522fdeb86; no voters:
I20260619 16:53:19.962229 18991 raft_consensus.cc:2781] T 965f7c212ae04e6fa5ca27b4899a9bcf P ba3c05bbb12247049fa503e05799f42b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:53:19.962402 18991 raft_consensus.cc:686] T 965f7c212ae04e6fa5ca27b4899a9bcf P ba3c05bbb12247049fa503e05799f42b [term 1 LEADER]: Becoming Leader. State: Replica: ba3c05bbb12247049fa503e05799f42b, State: Running, Role: LEADER
I20260619 16:53:19.962503 18991 consensus_queue.cc:237] T 965f7c212ae04e6fa5ca27b4899a9bcf P ba3c05bbb12247049fa503e05799f42b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86216cad23fd4146b9cd4782ed5b9acc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } } peers { permanent_uuid: "ba3c05bbb12247049fa503e05799f42b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39085 } } peers { permanent_uuid: "c07ecfa64b744fc18264b61522fdeb86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40637 } }
I20260619 16:53:19.963490 18509 catalog_manager.cc:5495] T 965f7c212ae04e6fa5ca27b4899a9bcf P ba3c05bbb12247049fa503e05799f42b reported cstate change: term changed from 0 to 1, leader changed from <none> to ba3c05bbb12247049fa503e05799f42b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ba3c05bbb12247049fa503e05799f42b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86216cad23fd4146b9cd4782ed5b9acc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba3c05bbb12247049fa503e05799f42b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c07ecfa64b744fc18264b61522fdeb86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40637 } health_report { overall_health: UNKNOWN } } }
I20260619 16:53:20.023046 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:20.020499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39182 (local address 127.0.98.3:39085)
0619 16:53:20.020584 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:20.020587 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:20.020595 (+ 8us) server_negotiation.cc:407] Connection header received
0619 16:53:20.020637 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:20.020637 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:20.020657 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:20.020722 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:53:20.020885 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:20.021426 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:20.022218 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:20.022418 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:20.022477 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:20.022648 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:53:20.022759 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:53:20.022859 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:20.022916 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 16:53:20.022926 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 16:53:20.025424 18788 raft_consensus.cc:1261] T 965f7c212ae04e6fa5ca27b4899a9bcf P c07ecfa64b744fc18264b61522fdeb86 [term 1 FOLLOWER]: Refusing update from remote peer ba3c05bbb12247049fa503e05799f42b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:53:20.025432 18654 raft_consensus.cc:1261] T 965f7c212ae04e6fa5ca27b4899a9bcf P 86216cad23fd4146b9cd4782ed5b9acc [term 1 FOLLOWER]: Refusing update from remote peer ba3c05bbb12247049fa503e05799f42b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:53:20.025839 18991 consensus_queue.cc:1035] T 965f7c212ae04e6fa5ca27b4899a9bcf P ba3c05bbb12247049fa503e05799f42b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07ecfa64b744fc18264b61522fdeb86" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:53:20.026003 18995 consensus_queue.cc:1035] T 965f7c212ae04e6fa5ca27b4899a9bcf P ba3c05bbb12247049fa503e05799f42b [LEADER]: Connected to new peer: Peer: permanent_uuid: "86216cad23fd4146b9cd4782ed5b9acc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:53:20.028733 18999 mvcc.cc:205] Tried to move back new op lower bound from 7298613248100499456 to 7298613247847071744. Current Snapshot: MvccSnapshot[applied={T|T < 7298613248100499456}]
I20260619 16:53:20.028896 18998 mvcc.cc:205] Tried to move back new op lower bound from 7298613248100499456 to 7298613247847071744. Current Snapshot: MvccSnapshot[applied={T|T < 7298613248100499456}]
I20260619 16:53:20.031802 18509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:53:20.031929 18509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:53:20.032864 18509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b400d2bf26c45829bcee774fa37b567: Sending DeleteTablet for 3 replicas of tablet 965f7c212ae04e6fa5ca27b4899a9bcf
I20260619 16:53:20.033202 18634 tablet_service.cc:1514] Processing DeleteTablet for tablet 965f7c212ae04e6fa5ca27b4899a9bcf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:53:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44274
I20260619 16:53:20.033246 18902 tablet_service.cc:1514] Processing DeleteTablet for tablet 965f7c212ae04e6fa5ca27b4899a9bcf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:53:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39174
I20260619 16:53:20.033457 19008 tablet_replica.cc:330] T 965f7c212ae04e6fa5ca27b4899a9bcf P 86216cad23fd4146b9cd4782ed5b9acc: stopping tablet replica
I20260619 16:53:20.033556 19009 tablet_replica.cc:330] T 965f7c212ae04e6fa5ca27b4899a9bcf P ba3c05bbb12247049fa503e05799f42b: stopping tablet replica
I20260619 16:53:20.033578 19008 raft_consensus.cc:2229] T 965f7c212ae04e6fa5ca27b4899a9bcf P 86216cad23fd4146b9cd4782ed5b9acc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:53:20.033586 18768 tablet_service.cc:1514] Processing DeleteTablet for tablet 965f7c212ae04e6fa5ca27b4899a9bcf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:53:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34700
I20260619 16:53:20.033704 19008 raft_consensus.cc:2258] T 965f7c212ae04e6fa5ca27b4899a9bcf P 86216cad23fd4146b9cd4782ed5b9acc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:53:20.033806 19009 raft_consensus.cc:2229] T 965f7c212ae04e6fa5ca27b4899a9bcf P ba3c05bbb12247049fa503e05799f42b [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:53:20.034039 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 18571
I20260619 16:53:20.033977 19009 raft_consensus.cc:2258] T 965f7c212ae04e6fa5ca27b4899a9bcf P ba3c05bbb12247049fa503e05799f42b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:53:20.036901 19009 ts_tablet_manager.cc:1905] T 965f7c212ae04e6fa5ca27b4899a9bcf P ba3c05bbb12247049fa503e05799f42b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:53:20.038841 19009 ts_tablet_manager.cc:1918] T 965f7c212ae04e6fa5ca27b4899a9bcf P ba3c05bbb12247049fa503e05799f42b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:53:20.038925 19009 log.cc:1196] T 965f7c212ae04e6fa5ca27b4899a9bcf P ba3c05bbb12247049fa503e05799f42b: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-2/wal/wals/965f7c212ae04e6fa5ca27b4899a9bcf
I20260619 16:53:20.039201 19009 ts_tablet_manager.cc:1939] T 965f7c212ae04e6fa5ca27b4899a9bcf P ba3c05bbb12247049fa503e05799f42b: Deleting consensus metadata
I20260619 16:53:20.039754 18496 catalog_manager.cc:4841] TS ba3c05bbb12247049fa503e05799f42b (127.0.98.3:39085): tablet 965f7c212ae04e6fa5ca27b4899a9bcf (table test-table [id=738679c734a841d6be6ff658800b51e9]) successfully deleted
W20260619 16:53:20.040681 18495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41101 (error 108)
I20260619 16:53:20.041076 19010 tablet_replica.cc:330] T 965f7c212ae04e6fa5ca27b4899a9bcf P c07ecfa64b744fc18264b61522fdeb86: stopping tablet replica
I20260619 16:53:20.041318 19010 raft_consensus.cc:2229] T 965f7c212ae04e6fa5ca27b4899a9bcf P c07ecfa64b744fc18264b61522fdeb86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:53:20.041402 18982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:20.041240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41101 (local address 127.0.0.1:44288)
0619 16:53:20.041304 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:20.041358 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 16:53:20.041512 19010 raft_consensus.cc:2258] T 965f7c212ae04e6fa5ca27b4899a9bcf P c07ecfa64b744fc18264b61522fdeb86 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 16:53:20.041535 18495 catalog_manager.cc:4584] TS 86216cad23fd4146b9cd4782ed5b9acc (127.0.98.1:41101): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 965f7c212ae04e6fa5ca27b4899a9bcf: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41101: connect: Connection refused (error 111)
I20260619 16:53:20.042030 19010 ts_tablet_manager.cc:1905] T 965f7c212ae04e6fa5ca27b4899a9bcf P c07ecfa64b744fc18264b61522fdeb86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:53:20.043784 19010 ts_tablet_manager.cc:1918] T 965f7c212ae04e6fa5ca27b4899a9bcf P c07ecfa64b744fc18264b61522fdeb86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:53:20.043862 19010 log.cc:1196] T 965f7c212ae04e6fa5ca27b4899a9bcf P c07ecfa64b744fc18264b61522fdeb86: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781887741903642-392-0/minicluster-data/ts-1/wal/wals/965f7c212ae04e6fa5ca27b4899a9bcf
I20260619 16:53:20.044104 19010 ts_tablet_manager.cc:1939] T 965f7c212ae04e6fa5ca27b4899a9bcf P c07ecfa64b744fc18264b61522fdeb86: Deleting consensus metadata
I20260619 16:53:20.044541 18496 catalog_manager.cc:4841] TS c07ecfa64b744fc18264b61522fdeb86 (127.0.98.2:40637): tablet 965f7c212ae04e6fa5ca27b4899a9bcf (table test-table [id=738679c734a841d6be6ff658800b51e9]) successfully deleted
I20260619 16:53:20.046423 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 18705
I20260619 16:53:20.052908 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 18839
I20260619 16:53:20.057614 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 18477
2026-06-19T16:53:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2200 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T16:53:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:53:20Z Disabled control of system clock
I20260619 16:53:20.087196 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35909
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:32941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:53:20.176728 19015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:20.177048 19015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:20.177073 19015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:20.178386 19015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:53:20.178426 19015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:20.178437 19015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:20.178448 19015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:53:20.178458 19015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:53:20.179860 19015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35909
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:20.180101 19015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:20.180400 19015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:20.185562 19023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:20.185704 19015 server_base.cc:1034] running on GCE node
W20260619 16:53:20.185817 19021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:20.185565 19020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:20.186106 19015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:20.186479 19015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:20.187639 19015 hybrid_clock.cc:648] HybridClock initialized: now 1781888000187624 us; error 36 us; skew 500 ppm
I20260619 16:53:20.189129 19015 webserver.cc:460] Webserver started at http://127.0.98.62:43177/ using document root <none> and password file <none>
I20260619 16:53:20.189442 19015 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:20.189499 19015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:20.189647 19015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:20.190945 19015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "00e4c845c4cd4392ba03a2bf4fe1b99e"
format_stamp: "Formatted at 2026-06-19 16:53:20 on dist-test-slave-jsw9"
server_key: "bd827df2688285775033e354e1f774f8"
server_key_iv: "1313ea783e13cffbec60318ba5398d21"
server_key_version: "encryptionkey@0"
I20260619 16:53:20.191362 19015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "00e4c845c4cd4392ba03a2bf4fe1b99e"
format_stamp: "Formatted at 2026-06-19 16:53:20 on dist-test-slave-jsw9"
server_key: "bd827df2688285775033e354e1f774f8"
server_key_iv: "1313ea783e13cffbec60318ba5398d21"
server_key_version: "encryptionkey@0"
I20260619 16:53:20.192847 19015 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 16:53:20.193778 19029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:20.193981 19015 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:53:20.194084 19015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "00e4c845c4cd4392ba03a2bf4fe1b99e"
format_stamp: "Formatted at 2026-06-19 16:53:20 on dist-test-slave-jsw9"
server_key: "bd827df2688285775033e354e1f774f8"
server_key_iv: "1313ea783e13cffbec60318ba5398d21"
server_key_version: "encryptionkey@0"
I20260619 16:53:20.194152 19015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:20.204483 19015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:20.204869 19015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:20.205017 19015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:20.209502 19015 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35909
I20260619 16:53:20.209537 19081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35909 every 8 connection(s)
I20260619 16:53:20.209911 19015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:53:20.211059 19082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:20.212667 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 19015
I20260619 16:53:20.212788 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:53:20.213024 392 external_mini_cluster.cc:1428] Setting key 97a857d842a8af5d7a19c97ecbdd5ed2
I20260619 16:53:20.214381 19082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00e4c845c4cd4392ba03a2bf4fe1b99e: Bootstrap starting.
I20260619 16:53:20.215173 19082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00e4c845c4cd4392ba03a2bf4fe1b99e: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:20.215473 19082 log.cc:824] T 00000000000000000000000000000000 P 00e4c845c4cd4392ba03a2bf4fe1b99e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:20.216174 19082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00e4c845c4cd4392ba03a2bf4fe1b99e: No bootstrap required, opened a new log
I20260619 16:53:20.218586 19082 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00e4c845c4cd4392ba03a2bf4fe1b99e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e4c845c4cd4392ba03a2bf4fe1b99e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35909 } }
I20260619 16:53:20.218739 19082 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00e4c845c4cd4392ba03a2bf4fe1b99e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:20.218771 19082 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00e4c845c4cd4392ba03a2bf4fe1b99e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00e4c845c4cd4392ba03a2bf4fe1b99e, State: Initialized, Role: FOLLOWER
I20260619 16:53:20.218900 19082 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00e4c845c4cd4392ba03a2bf4fe1b99e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e4c845c4cd4392ba03a2bf4fe1b99e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35909 } }
I20260619 16:53:20.219013 19082 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00e4c845c4cd4392ba03a2bf4fe1b99e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:53:20.219055 19082 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00e4c845c4cd4392ba03a2bf4fe1b99e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:53:20.219105 19082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00e4c845c4cd4392ba03a2bf4fe1b99e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:20.219326 19085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:20.213704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42456 (local address 127.0.98.62:35909)
0619 16:53:20.213933 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:20.213941 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:20.213969 (+ 28us) server_negotiation.cc:407] Connection header received
0619 16:53:20.214466 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:20.214475 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:20.214559 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:20.214697 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:20.214995 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:20.216004 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:20.217093 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:20.217574 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:20.217682 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:20.218466 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:20.218469 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:20.218470 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:20.218656 (+ 186us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:20.218694 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:20.218707 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:20.218768 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:20.218853 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 16:53:20.219113 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":102,"threads_started":1}
I20260619 16:53:20.219826 19082 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00e4c845c4cd4392ba03a2bf4fe1b99e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e4c845c4cd4392ba03a2bf4fe1b99e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35909 } }
I20260619 16:53:20.219949 19082 leader_election.cc:302] T 00000000000000000000000000000000 P 00e4c845c4cd4392ba03a2bf4fe1b99e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00e4c845c4cd4392ba03a2bf4fe1b99e; no voters:
I20260619 16:53:20.220122 19082 leader_election.cc:288] T 00000000000000000000000000000000 P 00e4c845c4cd4392ba03a2bf4fe1b99e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:53:20.220151 19087 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00e4c845c4cd4392ba03a2bf4fe1b99e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:53:20.220280 19087 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00e4c845c4cd4392ba03a2bf4fe1b99e [term 1 LEADER]: Becoming Leader. State: Replica: 00e4c845c4cd4392ba03a2bf4fe1b99e, State: Running, Role: LEADER
I20260619 16:53:20.220402 19082 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00e4c845c4cd4392ba03a2bf4fe1b99e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:53:20.220383 19087 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00e4c845c4cd4392ba03a2bf4fe1b99e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e4c845c4cd4392ba03a2bf4fe1b99e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35909 } }
I20260619 16:53:20.220820 19088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00e4c845c4cd4392ba03a2bf4fe1b99e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00e4c845c4cd4392ba03a2bf4fe1b99e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e4c845c4cd4392ba03a2bf4fe1b99e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35909 } } }
I20260619 16:53:20.220844 19089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00e4c845c4cd4392ba03a2bf4fe1b99e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00e4c845c4cd4392ba03a2bf4fe1b99e. Latest consensus state: current_term: 1 leader_uuid: "00e4c845c4cd4392ba03a2bf4fe1b99e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e4c845c4cd4392ba03a2bf4fe1b99e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35909 } } }
I20260619 16:53:20.221004 19089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00e4c845c4cd4392ba03a2bf4fe1b99e [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:20.221222 19096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:53:20.221395 19088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00e4c845c4cd4392ba03a2bf4fe1b99e [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:20.221794 19096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:53:20.223486 19096 catalog_manager.cc:1347] Generated new cluster ID: 4aa758b0c1484ad2afddfc76634a63dd
I20260619 16:53:20.223531 19096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:53:20.239960 19096 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:53:20.240590 19096 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:53:20.246078 19096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00e4c845c4cd4392ba03a2bf4fe1b99e: Generated new TSK 0
I20260619 16:53:20.246390 19096 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:53:20.253710 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35909
--builtin_ntp_servers=127.0.98.20:32941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:53:20.350585 19106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:20.350857 19106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:20.350891 19106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:20.352335 19106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:20.352397 19106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:20.352439 19106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:53:20.353955 19106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:20.354283 19106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:20.354637 19106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:20.360106 19114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:20.360163 19111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:20.360512 19112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:20.360664 19106 server_base.cc:1034] running on GCE node
I20260619 16:53:20.360901 19106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:20.361218 19106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:20.362375 19106 hybrid_clock.cc:648] HybridClock initialized: now 1781888000362367 us; error 59 us; skew 500 ppm
I20260619 16:53:20.364034 19106 webserver.cc:460] Webserver started at http://127.0.98.1:45361/ using document root <none> and password file <none>
I20260619 16:53:20.364319 19106 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:20.364378 19106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:20.364529 19106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:20.366130 19106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-0/data/instance:
uuid: "42d1a4ef8e5b421a8c663ccf7a952cf5"
format_stamp: "Formatted at 2026-06-19 16:53:20 on dist-test-slave-jsw9"
server_key: "a614451877c5c1e86835336ee7e31338"
server_key_iv: "f0098e898331e4c2bebe19cb09b2f002"
server_key_version: "encryptionkey@0"
I20260619 16:53:20.366636 19106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-0/wal/instance:
uuid: "42d1a4ef8e5b421a8c663ccf7a952cf5"
format_stamp: "Formatted at 2026-06-19 16:53:20 on dist-test-slave-jsw9"
server_key: "a614451877c5c1e86835336ee7e31338"
server_key_iv: "f0098e898331e4c2bebe19cb09b2f002"
server_key_version: "encryptionkey@0"
I20260619 16:53:20.368424 19106 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:53:20.369277 19120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:20.369498 19106 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:53:20.369602 19106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-0/wal
uuid: "42d1a4ef8e5b421a8c663ccf7a952cf5"
format_stamp: "Formatted at 2026-06-19 16:53:20 on dist-test-slave-jsw9"
server_key: "a614451877c5c1e86835336ee7e31338"
server_key_iv: "f0098e898331e4c2bebe19cb09b2f002"
server_key_version: "encryptionkey@0"
I20260619 16:53:20.369663 19106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:20.394232 19106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:20.394682 19106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:20.394806 19106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:20.395053 19106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:53:20.395408 19106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:53:20.395442 19106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:20.395467 19106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:53:20.395483 19106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:20.402155 19106 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43131
I20260619 16:53:20.402263 19233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43131 every 8 connection(s)
I20260619 16:53:20.402673 19106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:53:20.408636 19085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:20.403523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55119 (local address 127.0.98.62:35909)
0619 16:53:20.403596 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:20.403598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:20.403892 (+ 294us) server_negotiation.cc:407] Connection header received
0619 16:53:20.404463 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:20.404465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:20.404477 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:20.404542 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:20.405359 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:20.405924 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:20.407066 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:20.407277 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:20.407357 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:20.408390 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:20.408391 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:20.408392 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:20.408397 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:20.408415 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:20.408419 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:20.408465 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:20.408534 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 16:53:20.408561 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 16:53:20.408865 19236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:20.403502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35909 (local address 127.0.98.1:55119)
0619 16:53:20.403724 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:20.403772 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:20.404367 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:20.404636 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:20.404640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:20.404650 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:20.405228 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:20.405233 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:20.406022 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:20.406022 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:20.406982 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:20.406985 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:20.407395 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:20.407395 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:20.407538 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:20.408188 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:53:20.408189 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:20.408224 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:53:20.408251 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:20.408472 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:20.408476 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:20.408479 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:53:20.408532 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 16:53:20.408629 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":91,"threads_started":1}
I20260619 16:53:20.409395 19234 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35909
I20260619 16:53:20.409495 19234 heartbeater.cc:464] Registering TS with master...
I20260619 16:53:20.409669 19234 heartbeater.cc:511] Master 127.0.98.62:35909 requested a full tablet report, sending...
I20260619 16:53:20.410348 19046 ts_manager.cc:194] Registered new tserver with Master: 42d1a4ef8e5b421a8c663ccf7a952cf5 (127.0.98.1:43131)
I20260619 16:53:20.411046 19046 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:55119
I20260619 16:53:20.412577 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 19106
I20260619 16:53:20.412683 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:53:20.412875 392 external_mini_cluster.cc:1428] Setting key 8c3e6f325defebc2421f1944cdc93912
I20260619 16:53:20.413929 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35909
--builtin_ntp_servers=127.0.98.20:32941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:53:20.507426 19237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:20.507691 19237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:20.507719 19237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:20.509059 19237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:20.509109 19237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:20.509140 19237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:53:20.510697 19237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:20.510970 19237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:20.511286 19237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:20.517153 19242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:20.517139 19245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:20.517133 19243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:20.517485 19237 server_base.cc:1034] running on GCE node
I20260619 16:53:20.517818 19237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:20.518076 19237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:20.519244 19237 hybrid_clock.cc:648] HybridClock initialized: now 1781888000519241 us; error 46 us; skew 500 ppm
I20260619 16:53:20.520982 19237 webserver.cc:460] Webserver started at http://127.0.98.2:44633/ using document root <none> and password file <none>
I20260619 16:53:20.521324 19237 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:20.521384 19237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:20.521502 19237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:20.522773 19237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-1/data/instance:
uuid: "96fa63cbf0c84edfaf66ae55042bd8df"
format_stamp: "Formatted at 2026-06-19 16:53:20 on dist-test-slave-jsw9"
server_key: "a90b719d0486527c4d9faf206680f669"
server_key_iv: "1fa92e1fe6d73127e6e26299d8c91aca"
server_key_version: "encryptionkey@0"
I20260619 16:53:20.523110 19237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-1/wal/instance:
uuid: "96fa63cbf0c84edfaf66ae55042bd8df"
format_stamp: "Formatted at 2026-06-19 16:53:20 on dist-test-slave-jsw9"
server_key: "a90b719d0486527c4d9faf206680f669"
server_key_iv: "1fa92e1fe6d73127e6e26299d8c91aca"
server_key_version: "encryptionkey@0"
I20260619 16:53:20.524520 19237 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 16:53:20.525406 19251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:20.525611 19237 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:53:20.525702 19237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-1/wal
uuid: "96fa63cbf0c84edfaf66ae55042bd8df"
format_stamp: "Formatted at 2026-06-19 16:53:20 on dist-test-slave-jsw9"
server_key: "a90b719d0486527c4d9faf206680f669"
server_key_iv: "1fa92e1fe6d73127e6e26299d8c91aca"
server_key_version: "encryptionkey@0"
I20260619 16:53:20.525768 19237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:20.543247 19237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:20.543635 19237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:20.543906 19237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:20.544204 19237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:53:20.544595 19237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:53:20.544631 19237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:20.544657 19237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:53:20.544668 19237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:20.551545 19237 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38455
I20260619 16:53:20.551626 19364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38455 every 8 connection(s)
I20260619 16:53:20.552018 19237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:53:20.558549 19085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:20.552953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45181 (local address 127.0.98.62:35909)
0619 16:53:20.553038 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:20.553041 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:20.553215 (+ 174us) server_negotiation.cc:407] Connection header received
0619 16:53:20.553809 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:20.553810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:20.553826 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:20.553903 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:20.555023 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:20.555616 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:20.556943 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:20.557163 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:20.557228 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:20.558152 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:20.558153 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:20.558154 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:20.558160 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:20.558179 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:20.558212 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:20.558264 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:20.558440 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 16:53:20.558470 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:53:20.558692 19367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:20.552837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35909 (local address 127.0.98.2:45181)
0619 16:53:20.553043 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:20.553085 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:20.553682 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:20.554033 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:20.554039 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:20.554053 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:20.554910 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:20.554919 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:20.555733 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:20.555735 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:20.556833 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:20.556837 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:20.557242 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:20.557243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:20.557374 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:20.557989 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:53:20.557989 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:20.558023 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:53:20.558052 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:20.558292 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:20.558298 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:20.558301 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:53:20.558379 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 16:53:20.558481 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
I20260619 16:53:20.559293 19365 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35909
I20260619 16:53:20.559396 19365 heartbeater.cc:464] Registering TS with master...
I20260619 16:53:20.559573 19365 heartbeater.cc:511] Master 127.0.98.62:35909 requested a full tablet report, sending...
I20260619 16:53:20.560019 19046 ts_manager.cc:194] Registered new tserver with Master: 96fa63cbf0c84edfaf66ae55042bd8df (127.0.98.2:38455)
I20260619 16:53:20.560257 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 19237
I20260619 16:53:20.560362 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:53:20.560585 392 external_mini_cluster.cc:1428] Setting key 83215bb72eac785667b5850a4caadc43
I20260619 16:53:20.560583 19046 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:45181
I20260619 16:53:20.561546 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35909
--builtin_ntp_servers=127.0.98.20:32941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:53:20.660059 19368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:20.660328 19368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:20.660359 19368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:20.661628 19368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:20.661669 19368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:20.661695 19368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:53:20.663158 19368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:20.663467 19368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:20.663798 19368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:20.669523 19373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:20.669523 19374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:20.669524 19376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:20.669826 19368 server_base.cc:1034] running on GCE node
I20260619 16:53:20.670050 19368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:20.670365 19368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:20.671674 19368 hybrid_clock.cc:648] HybridClock initialized: now 1781888000671632 us; error 38 us; skew 500 ppm
I20260619 16:53:20.674171 19368 webserver.cc:460] Webserver started at http://127.0.98.3:44303/ using document root <none> and password file <none>
I20260619 16:53:20.674571 19368 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:20.674654 19368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:20.674813 19368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:20.676112 19368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-2/data/instance:
uuid: "ececadece1c64419b1d4e8050c3c1fc5"
format_stamp: "Formatted at 2026-06-19 16:53:20 on dist-test-slave-jsw9"
server_key: "48e2802d83b0e6a68fdc710c7985885a"
server_key_iv: "0661d30fce842fbeb07a61ad8d10df08"
server_key_version: "encryptionkey@0"
I20260619 16:53:20.676507 19368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ececadece1c64419b1d4e8050c3c1fc5"
format_stamp: "Formatted at 2026-06-19 16:53:20 on dist-test-slave-jsw9"
server_key: "48e2802d83b0e6a68fdc710c7985885a"
server_key_iv: "0661d30fce842fbeb07a61ad8d10df08"
server_key_version: "encryptionkey@0"
I20260619 16:53:20.677949 19368 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 16:53:20.678967 19382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:20.679622 19368 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:53:20.679728 19368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-2/wal
uuid: "ececadece1c64419b1d4e8050c3c1fc5"
format_stamp: "Formatted at 2026-06-19 16:53:20 on dist-test-slave-jsw9"
server_key: "48e2802d83b0e6a68fdc710c7985885a"
server_key_iv: "0661d30fce842fbeb07a61ad8d10df08"
server_key_version: "encryptionkey@0"
I20260619 16:53:20.679811 19368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:20.705750 19368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:20.706274 19368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:20.706450 19368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:20.706732 19368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:53:20.707118 19368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:53:20.707258 19368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:20.707314 19368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:53:20.707340 19368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:20.713992 19368 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45465
I20260619 16:53:20.714108 19495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45465 every 8 connection(s)
I20260619 16:53:20.714546 19368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:53:20.718333 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 19368
I20260619 16:53:20.718663 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:53:20.719025 392 external_mini_cluster.cc:1428] Setting key 62c8aa07a99acc8ca5f65b2653afa270
I20260619 16:53:20.721575 19085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:20.715493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42005 (local address 127.0.98.62:35909)
0619 16:53:20.715601 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:20.715604 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:20.715938 (+ 334us) server_negotiation.cc:407] Connection header received
0619 16:53:20.716468 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:20.716470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:20.716481 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:20.716535 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:20.717300 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:20.717866 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:20.719895 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:20.720192 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:20.720265 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:20.721244 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:20.721245 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:20.721245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:20.721250 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:20.721280 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:20.721285 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:20.721346 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:20.721420 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 16:53:20.721455 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 16:53:20.721794 19498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:20.715473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35909 (local address 127.0.98.3:42005)
0619 16:53:20.715718 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:20.715765 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:20.716358 (+ 593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:20.716605 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:20.716609 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:20.716619 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:20.717195 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:20.717200 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:20.717999 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:20.718001 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:20.719818 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:20.719822 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:20.720295 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:20.720297 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:20.720483 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:20.721074 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:53:20.721075 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:20.721112 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:53:20.721140 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:20.721362 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:20.721366 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:20.721370 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:53:20.721421 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 16:53:20.721545 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":144,"threads_started":1}
I20260619 16:53:20.722347 19496 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35909
I20260619 16:53:20.722445 19496 heartbeater.cc:464] Registering TS with master...
I20260619 16:53:20.722613 19496 heartbeater.cc:511] Master 127.0.98.62:35909 requested a full tablet report, sending...
I20260619 16:53:20.722960 19046 ts_manager.cc:194] Registered new tserver with Master: ececadece1c64419b1d4e8050c3c1fc5 (127.0.98.3:45465)
I20260619 16:53:20.723347 19046 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:42005
I20260619 16:53:20.731544 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:53:20.736425 19085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:20.733347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47684 (local address 127.0.98.62:35909)
0619 16:53:20.733434 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:20.733436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:20.733462 (+ 26us) server_negotiation.cc:407] Connection header received
0619 16:53:20.733562 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:20.733563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:20.733575 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:20.733619 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:20.733781 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:20.734384 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:20.735136 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:20.735330 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:20.735394 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:20.736136 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:20.736137 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:20.736138 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:20.736142 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:20.736158 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:20.736161 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:20.736237 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:20.736317 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 16:53:20.736330 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 16:53:20.738435 19046 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:53:20.740446 19046 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:53:20.747438 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:20.742920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45465 (local address 127.0.0.1:60124)
0619 16:53:20.743334 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:20.743342 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:20.743374 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:20.743741 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:20.743742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:20.743745 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:20.743807 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:20.743809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:20.745146 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:20.745147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:20.746264 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:20.746266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:20.747076 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:20.747077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:20.747162 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:20.747177 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 16:53:20.747248 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 16:53:20.747316 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":38,"spinlock_wait_cycles":768,"thread_start_us":77,"threads_started":1}
I20260619 16:53:20.747514 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:20.742919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43131 (local address 127.0.0.1:58666)
0619 16:53:20.743135 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:20.743183 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:20.743251 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:20.743567 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:20.743571 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:20.743574 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:20.743673 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:20.743676 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:20.744780 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:20.744781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:20.746088 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:20.746091 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:20.747264 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:20.747265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:20.747320 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:20.747324 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:53:20.747371 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 16:53:20.747408 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":79,"threads_started":1}
I20260619 16:53:20.747572 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:20.743010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58666 (local address 127.0.98.1:43131)
0619 16:53:20.743290 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:20.743296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:20.743310 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:53:20.743360 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:20.743362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:20.743380 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:20.743473 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:20.743761 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:20.744706 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:20.746204 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:20.747168 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:20.747263 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:20.747361 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:20.747393 (+ 32us) server_negotiation.cc:299] Negotiation successful
0619 16:53:20.747473 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":115,"threads_started":1}
I20260619 16:53:20.747859 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:20.743240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60124 (local address 127.0.98.3:45465)
0619 16:53:20.743494 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:20.743498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:20.743509 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:53:20.743552 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:20.743554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:20.743571 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:20.743653 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:20.743882 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:20.745050 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:20.746350 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:20.746994 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:20.747067 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:20.747182 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:20.747695 (+ 513us) server_negotiation.cc:299] Negotiation successful
0619 16:53:20.747755 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":146,"threads_started":1}
I20260619 16:53:20.748241 19507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:20.743139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38455 (local address 127.0.0.1:49712)
0619 16:53:20.743716 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:20.743725 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:20.743771 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:20.744101 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:20.744102 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:20.744105 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:20.744154 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:20.744155 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:20.745349 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:20.745350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:20.746277 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:20.746281 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:20.747452 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:20.747454 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:20.747514 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:20.747519 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:53:20.748127 (+ 608us) client_negotiation.cc:240] Negotiation successful
0619 16:53:20.748183 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":48,"threads_started":1}
I20260619 16:53:20.748669 19509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:20.743498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49712 (local address 127.0.98.2:38455)
0619 16:53:20.743928 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:20.743933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:20.743942 (+ 9us) server_negotiation.cc:407] Connection header received
0619 16:53:20.743986 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:20.743988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:20.744002 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:20.744090 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:20.744216 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:20.745273 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:20.746381 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:20.747345 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:20.748422 (+ 1077us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:20.748534 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:20.748557 (+ 23us) server_negotiation.cc:299] Negotiation successful
0619 16:53:20.748598 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":114,"threads_started":1}
I20260619 16:53:20.748540 19168 tablet_service.cc:1467] Processing CreateTablet for tablet 7daa38aa4e564338a47e58898086383c (DEFAULT_TABLE table=test-table [id=9102cecbef0d44768239395dde14b83a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:53:20.748960 19168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7daa38aa4e564338a47e58898086383c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:20.749310 19299 tablet_service.cc:1467] Processing CreateTablet for tablet 7daa38aa4e564338a47e58898086383c (DEFAULT_TABLE table=test-table [id=9102cecbef0d44768239395dde14b83a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:53:20.749624 19299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7daa38aa4e564338a47e58898086383c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:20.751008 19430 tablet_service.cc:1467] Processing CreateTablet for tablet 7daa38aa4e564338a47e58898086383c (DEFAULT_TABLE table=test-table [id=9102cecbef0d44768239395dde14b83a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:53:20.751422 19430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7daa38aa4e564338a47e58898086383c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:20.752417 19510 tablet_bootstrap.cc:492] T 7daa38aa4e564338a47e58898086383c P 42d1a4ef8e5b421a8c663ccf7a952cf5: Bootstrap starting.
I20260619 16:53:20.753041 19511 tablet_bootstrap.cc:492] T 7daa38aa4e564338a47e58898086383c P 96fa63cbf0c84edfaf66ae55042bd8df: Bootstrap starting.
I20260619 16:53:20.753273 19510 tablet_bootstrap.cc:654] T 7daa38aa4e564338a47e58898086383c P 42d1a4ef8e5b421a8c663ccf7a952cf5: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:20.753649 19510 log.cc:824] T 7daa38aa4e564338a47e58898086383c P 42d1a4ef8e5b421a8c663ccf7a952cf5: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:20.753753 19511 tablet_bootstrap.cc:654] T 7daa38aa4e564338a47e58898086383c P 96fa63cbf0c84edfaf66ae55042bd8df: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:20.754046 19511 log.cc:824] T 7daa38aa4e564338a47e58898086383c P 96fa63cbf0c84edfaf66ae55042bd8df: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:20.754678 19510 tablet_bootstrap.cc:492] T 7daa38aa4e564338a47e58898086383c P 42d1a4ef8e5b421a8c663ccf7a952cf5: No bootstrap required, opened a new log
I20260619 16:53:20.754758 19510 ts_tablet_manager.cc:1397] T 7daa38aa4e564338a47e58898086383c P 42d1a4ef8e5b421a8c663ccf7a952cf5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:53:20.755029 19514 tablet_bootstrap.cc:492] T 7daa38aa4e564338a47e58898086383c P ececadece1c64419b1d4e8050c3c1fc5: Bootstrap starting.
I20260619 16:53:20.755666 19514 tablet_bootstrap.cc:654] T 7daa38aa4e564338a47e58898086383c P ececadece1c64419b1d4e8050c3c1fc5: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:20.755944 19514 log.cc:824] T 7daa38aa4e564338a47e58898086383c P ececadece1c64419b1d4e8050c3c1fc5: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:20.756650 19514 tablet_bootstrap.cc:492] T 7daa38aa4e564338a47e58898086383c P ececadece1c64419b1d4e8050c3c1fc5: No bootstrap required, opened a new log
I20260619 16:53:20.756700 19514 ts_tablet_manager.cc:1397] T 7daa38aa4e564338a47e58898086383c P ececadece1c64419b1d4e8050c3c1fc5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:53:20.757135 19510 raft_consensus.cc:348] T 7daa38aa4e564338a47e58898086383c P 42d1a4ef8e5b421a8c663ccf7a952cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42d1a4ef8e5b421a8c663ccf7a952cf5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43131 } } peers { permanent_uuid: "96fa63cbf0c84edfaf66ae55042bd8df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38455 } } peers { permanent_uuid: "ececadece1c64419b1d4e8050c3c1fc5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45465 } }
I20260619 16:53:20.757289 19510 raft_consensus.cc:374] T 7daa38aa4e564338a47e58898086383c P 42d1a4ef8e5b421a8c663ccf7a952cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:20.757323 19510 raft_consensus.cc:729] T 7daa38aa4e564338a47e58898086383c P 42d1a4ef8e5b421a8c663ccf7a952cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42d1a4ef8e5b421a8c663ccf7a952cf5, State: Initialized, Role: FOLLOWER
I20260619 16:53:20.757452 19510 consensus_queue.cc:260] T 7daa38aa4e564338a47e58898086383c P 42d1a4ef8e5b421a8c663ccf7a952cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42d1a4ef8e5b421a8c663ccf7a952cf5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43131 } } peers { permanent_uuid: "96fa63cbf0c84edfaf66ae55042bd8df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38455 } } peers { permanent_uuid: "ececadece1c64419b1d4e8050c3c1fc5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45465 } }
I20260619 16:53:20.757746 19510 ts_tablet_manager.cc:1428] T 7daa38aa4e564338a47e58898086383c P 42d1a4ef8e5b421a8c663ccf7a952cf5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 16:53:20.757799 19234 heartbeater.cc:503] Master 127.0.98.62:35909 was elected leader, sending a full tablet report...
I20260619 16:53:20.758543 19511 tablet_bootstrap.cc:492] T 7daa38aa4e564338a47e58898086383c P 96fa63cbf0c84edfaf66ae55042bd8df: No bootstrap required, opened a new log
I20260619 16:53:20.758623 19511 ts_tablet_manager.cc:1397] T 7daa38aa4e564338a47e58898086383c P 96fa63cbf0c84edfaf66ae55042bd8df: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260619 16:53:20.759425 19514 raft_consensus.cc:348] T 7daa38aa4e564338a47e58898086383c P ececadece1c64419b1d4e8050c3c1fc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42d1a4ef8e5b421a8c663ccf7a952cf5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43131 } } peers { permanent_uuid: "96fa63cbf0c84edfaf66ae55042bd8df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38455 } } peers { permanent_uuid: "ececadece1c64419b1d4e8050c3c1fc5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45465 } }
I20260619 16:53:20.759611 19514 raft_consensus.cc:374] T 7daa38aa4e564338a47e58898086383c P ececadece1c64419b1d4e8050c3c1fc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:20.759653 19514 raft_consensus.cc:729] T 7daa38aa4e564338a47e58898086383c P ececadece1c64419b1d4e8050c3c1fc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ececadece1c64419b1d4e8050c3c1fc5, State: Initialized, Role: FOLLOWER
I20260619 16:53:20.759774 19514 consensus_queue.cc:260] T 7daa38aa4e564338a47e58898086383c P ececadece1c64419b1d4e8050c3c1fc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42d1a4ef8e5b421a8c663ccf7a952cf5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43131 } } peers { permanent_uuid: "96fa63cbf0c84edfaf66ae55042bd8df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38455 } } peers { permanent_uuid: "ececadece1c64419b1d4e8050c3c1fc5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45465 } }
I20260619 16:53:20.760066 19514 ts_tablet_manager.cc:1428] T 7daa38aa4e564338a47e58898086383c P ececadece1c64419b1d4e8050c3c1fc5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 16:53:20.760100 19496 heartbeater.cc:503] Master 127.0.98.62:35909 was elected leader, sending a full tablet report...
I20260619 16:53:20.761317 19511 raft_consensus.cc:348] T 7daa38aa4e564338a47e58898086383c P 96fa63cbf0c84edfaf66ae55042bd8df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42d1a4ef8e5b421a8c663ccf7a952cf5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43131 } } peers { permanent_uuid: "96fa63cbf0c84edfaf66ae55042bd8df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38455 } } peers { permanent_uuid: "ececadece1c64419b1d4e8050c3c1fc5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45465 } }
I20260619 16:53:20.761502 19511 raft_consensus.cc:374] T 7daa38aa4e564338a47e58898086383c P 96fa63cbf0c84edfaf66ae55042bd8df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:20.761526 19511 raft_consensus.cc:729] T 7daa38aa4e564338a47e58898086383c P 96fa63cbf0c84edfaf66ae55042bd8df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96fa63cbf0c84edfaf66ae55042bd8df, State: Initialized, Role: FOLLOWER
I20260619 16:53:20.761632 19511 consensus_queue.cc:260] T 7daa38aa4e564338a47e58898086383c P 96fa63cbf0c84edfaf66ae55042bd8df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42d1a4ef8e5b421a8c663ccf7a952cf5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43131 } } peers { permanent_uuid: "96fa63cbf0c84edfaf66ae55042bd8df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38455 } } peers { permanent_uuid: "ececadece1c64419b1d4e8050c3c1fc5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45465 } }
I20260619 16:53:20.761864 19511 ts_tablet_manager.cc:1428] T 7daa38aa4e564338a47e58898086383c P 96fa63cbf0c84edfaf66ae55042bd8df: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260619 16:53:20.761924 19365 heartbeater.cc:503] Master 127.0.98.62:35909 was elected leader, sending a full tablet report...
W20260619 16:53:20.802668 19366 tablet.cc:2369] T 7daa38aa4e564338a47e58898086383c P 96fa63cbf0c84edfaf66ae55042bd8df: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:53:20.903626 19235 tablet.cc:2369] T 7daa38aa4e564338a47e58898086383c P 42d1a4ef8e5b421a8c663ccf7a952cf5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:53:20.965323 19497 tablet.cc:2369] T 7daa38aa4e564338a47e58898086383c P ececadece1c64419b1d4e8050c3c1fc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:53:20.981122 19517 raft_consensus.cc:482] T 7daa38aa4e564338a47e58898086383c P ececadece1c64419b1d4e8050c3c1fc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:53:20.981302 19517 raft_consensus.cc:504] T 7daa38aa4e564338a47e58898086383c P ececadece1c64419b1d4e8050c3c1fc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42d1a4ef8e5b421a8c663ccf7a952cf5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43131 } } peers { permanent_uuid: "96fa63cbf0c84edfaf66ae55042bd8df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38455 } } peers { permanent_uuid: "ececadece1c64419b1d4e8050c3c1fc5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45465 } }
I20260619 16:53:20.981832 19517 leader_election.cc:288] T 7daa38aa4e564338a47e58898086383c P ececadece1c64419b1d4e8050c3c1fc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42d1a4ef8e5b421a8c663ccf7a952cf5 (127.0.98.1:43131), 96fa63cbf0c84edfaf66ae55042bd8df (127.0.98.2:38455)
I20260619 16:53:20.985473 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:20.982385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48083 (local address 127.0.98.1:43131)
0619 16:53:20.982440 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:20.982443 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:20.982452 (+ 9us) server_negotiation.cc:407] Connection header received
0619 16:53:20.982488 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:20.982489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:20.982504 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:20.982566 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:20.982811 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:20.983633 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:20.984715 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:20.985111 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:20.985178 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:20.985192 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:20.985327 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 16:53:20.985375 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 16:53:20.985502 19498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:20.982163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43131 (local address 127.0.98.3:48083)
0619 16:53:20.982251 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:20.982267 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:20.982340 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:20.982627 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:20.982628 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:20.982631 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:20.982721 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:20.982724 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:20.983756 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:20.983757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:20.984615 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:20.984619 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:20.985192 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:20.985193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:20.985265 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:20.985268 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:53:20.985315 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 16:53:20.985374 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 16:53:20.985495 19509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:20.982513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38301 (local address 127.0.98.2:38455)
0619 16:53:20.982568 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:20.982570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:20.982699 (+ 129us) server_negotiation.cc:407] Connection header received
0619 16:53:20.982757 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:20.982758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:20.982772 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:20.982871 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:20.983073 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:20.983803 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:20.984749 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:20.985145 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:20.985204 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:20.985215 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:20.985368 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 16:53:20.985418 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 16:53:20.985513 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:20.982358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38455 (local address 127.0.98.3:38301)
0619 16:53:20.982627 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:20.982636 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:20.982695 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:20.982900 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:20.982902 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:20.982905 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:20.982996 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:20.982998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:20.983874 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:20.983875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:20.984655 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:20.984657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:20.985217 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:20.985218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:20.985295 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:20.985298 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:53:20.985346 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 16:53:20.985388 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":179,"threads_started":1}
I20260619 16:53:20.985951 19188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7daa38aa4e564338a47e58898086383c" candidate_uuid: "ececadece1c64419b1d4e8050c3c1fc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42d1a4ef8e5b421a8c663ccf7a952cf5" is_pre_election: true
I20260619 16:53:20.985915 19319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7daa38aa4e564338a47e58898086383c" candidate_uuid: "ececadece1c64419b1d4e8050c3c1fc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96fa63cbf0c84edfaf66ae55042bd8df" is_pre_election: true
I20260619 16:53:20.986169 19188 raft_consensus.cc:2454] T 7daa38aa4e564338a47e58898086383c P 42d1a4ef8e5b421a8c663ccf7a952cf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ececadece1c64419b1d4e8050c3c1fc5 in term 0.
I20260619 16:53:20.986246 19319 raft_consensus.cc:2454] T 7daa38aa4e564338a47e58898086383c P 96fa63cbf0c84edfaf66ae55042bd8df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ececadece1c64419b1d4e8050c3c1fc5 in term 0.
I20260619 16:53:20.986493 19386 leader_election.cc:302] T 7daa38aa4e564338a47e58898086383c P ececadece1c64419b1d4e8050c3c1fc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42d1a4ef8e5b421a8c663ccf7a952cf5, ececadece1c64419b1d4e8050c3c1fc5; no voters:
I20260619 16:53:20.986629 19517 raft_consensus.cc:2781] T 7daa38aa4e564338a47e58898086383c P ececadece1c64419b1d4e8050c3c1fc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:53:20.986701 19517 raft_consensus.cc:482] T 7daa38aa4e564338a47e58898086383c P ececadece1c64419b1d4e8050c3c1fc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:53:20.986726 19517 raft_consensus.cc:3037] T 7daa38aa4e564338a47e58898086383c P ececadece1c64419b1d4e8050c3c1fc5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:20.987789 19517 raft_consensus.cc:504] T 7daa38aa4e564338a47e58898086383c P ececadece1c64419b1d4e8050c3c1fc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42d1a4ef8e5b421a8c663ccf7a952cf5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43131 } } peers { permanent_uuid: "96fa63cbf0c84edfaf66ae55042bd8df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38455 } } peers { permanent_uuid: "ececadece1c64419b1d4e8050c3c1fc5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45465 } }
I20260619 16:53:20.987973 19517 leader_election.cc:288] T 7daa38aa4e564338a47e58898086383c P ececadece1c64419b1d4e8050c3c1fc5 [CANDIDATE]: Term 1 election: Requested vote from peers 42d1a4ef8e5b421a8c663ccf7a952cf5 (127.0.98.1:43131), 96fa63cbf0c84edfaf66ae55042bd8df (127.0.98.2:38455)
I20260619 16:53:20.988261 19319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7daa38aa4e564338a47e58898086383c" candidate_uuid: "ececadece1c64419b1d4e8050c3c1fc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96fa63cbf0c84edfaf66ae55042bd8df"
I20260619 16:53:20.988286 19188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7daa38aa4e564338a47e58898086383c" candidate_uuid: "ececadece1c64419b1d4e8050c3c1fc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42d1a4ef8e5b421a8c663ccf7a952cf5"
I20260619 16:53:20.988411 19188 raft_consensus.cc:3037] T 7daa38aa4e564338a47e58898086383c P 42d1a4ef8e5b421a8c663ccf7a952cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:20.988411 19319 raft_consensus.cc:3037] T 7daa38aa4e564338a47e58898086383c P 96fa63cbf0c84edfaf66ae55042bd8df [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:20.989566 19319 raft_consensus.cc:2454] T 7daa38aa4e564338a47e58898086383c P 96fa63cbf0c84edfaf66ae55042bd8df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ececadece1c64419b1d4e8050c3c1fc5 in term 1.
I20260619 16:53:20.989566 19188 raft_consensus.cc:2454] T 7daa38aa4e564338a47e58898086383c P 42d1a4ef8e5b421a8c663ccf7a952cf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ececadece1c64419b1d4e8050c3c1fc5 in term 1.
I20260619 16:53:20.989841 19386 leader_election.cc:302] T 7daa38aa4e564338a47e58898086383c P ececadece1c64419b1d4e8050c3c1fc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42d1a4ef8e5b421a8c663ccf7a952cf5, ececadece1c64419b1d4e8050c3c1fc5; no voters:
I20260619 16:53:20.990010 19517 raft_consensus.cc:2781] T 7daa38aa4e564338a47e58898086383c P ececadece1c64419b1d4e8050c3c1fc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:53:20.990406 19517 raft_consensus.cc:686] T 7daa38aa4e564338a47e58898086383c P ececadece1c64419b1d4e8050c3c1fc5 [term 1 LEADER]: Becoming Leader. State: Replica: ececadece1c64419b1d4e8050c3c1fc5, State: Running, Role: LEADER
I20260619 16:53:20.990641 19517 consensus_queue.cc:237] T 7daa38aa4e564338a47e58898086383c P ececadece1c64419b1d4e8050c3c1fc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42d1a4ef8e5b421a8c663ccf7a952cf5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43131 } } peers { permanent_uuid: "96fa63cbf0c84edfaf66ae55042bd8df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38455 } } peers { permanent_uuid: "ececadece1c64419b1d4e8050c3c1fc5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45465 } }
I20260619 16:53:20.992005 19045 catalog_manager.cc:5495] T 7daa38aa4e564338a47e58898086383c P ececadece1c64419b1d4e8050c3c1fc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to ececadece1c64419b1d4e8050c3c1fc5 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ececadece1c64419b1d4e8050c3c1fc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42d1a4ef8e5b421a8c663ccf7a952cf5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96fa63cbf0c84edfaf66ae55042bd8df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ececadece1c64419b1d4e8050c3c1fc5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45465 } health_report { overall_health: HEALTHY } } }
I20260619 16:53:21.037750 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:21.034728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60138 (local address 127.0.98.3:45465)
0619 16:53:21.034799 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:21.034801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:21.034810 (+ 9us) server_negotiation.cc:407] Connection header received
0619 16:53:21.034856 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:21.034856 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:21.034882 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:21.034951 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:53:21.035104 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:21.035628 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:21.036728 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:21.036963 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:21.037034 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:21.037222 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:53:21.037335 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:53:21.037435 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:21.037607 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 16:53:21.037621 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 16:53:21.040455 19188 raft_consensus.cc:1261] T 7daa38aa4e564338a47e58898086383c P 42d1a4ef8e5b421a8c663ccf7a952cf5 [term 1 FOLLOWER]: Refusing update from remote peer ececadece1c64419b1d4e8050c3c1fc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:53:21.040454 19319 raft_consensus.cc:1261] T 7daa38aa4e564338a47e58898086383c P 96fa63cbf0c84edfaf66ae55042bd8df [term 1 FOLLOWER]: Refusing update from remote peer ececadece1c64419b1d4e8050c3c1fc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:53:21.040904 19517 consensus_queue.cc:1035] T 7daa38aa4e564338a47e58898086383c P ececadece1c64419b1d4e8050c3c1fc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42d1a4ef8e5b421a8c663ccf7a952cf5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:53:21.041069 19520 consensus_queue.cc:1035] T 7daa38aa4e564338a47e58898086383c P ececadece1c64419b1d4e8050c3c1fc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fa63cbf0c84edfaf66ae55042bd8df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:53:21.043704 19523 mvcc.cc:205] Tried to move back new op lower bound from 7298613252257398784 to 7298613252058529792. Current Snapshot: MvccSnapshot[applied={T|T < 7298613252257398784}]
I20260619 16:53:21.045276 19525 mvcc.cc:205] Tried to move back new op lower bound from 7298613252257398784 to 7298613252058529792. Current Snapshot: MvccSnapshot[applied={T|T < 7298613252257398784 or (T in {7298613252257398784})}]
I20260619 16:53:21.046984 19045 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:53:21.047102 19045 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:53:21.048051 19045 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00e4c845c4cd4392ba03a2bf4fe1b99e: Sending DeleteTablet for 3 replicas of tablet 7daa38aa4e564338a47e58898086383c
I20260619 16:53:21.048444 19430 tablet_service.cc:1514] Processing DeleteTablet for tablet 7daa38aa4e564338a47e58898086383c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:53:21 UTC) from {username='slave'} at 127.0.0.1:60124
I20260619 16:53:21.048455 19168 tablet_service.cc:1514] Processing DeleteTablet for tablet 7daa38aa4e564338a47e58898086383c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:53:21 UTC) from {username='slave'} at 127.0.0.1:58666
I20260619 16:53:21.048534 19299 tablet_service.cc:1514] Processing DeleteTablet for tablet 7daa38aa4e564338a47e58898086383c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:53:21 UTC) from {username='slave'} at 127.0.0.1:49712
I20260619 16:53:21.048722 19533 tablet_replica.cc:330] T 7daa38aa4e564338a47e58898086383c P 42d1a4ef8e5b421a8c663ccf7a952cf5: stopping tablet replica
I20260619 16:53:21.048833 19534 tablet_replica.cc:330] T 7daa38aa4e564338a47e58898086383c P ececadece1c64419b1d4e8050c3c1fc5: stopping tablet replica
I20260619 16:53:21.048851 19533 raft_consensus.cc:2229] T 7daa38aa4e564338a47e58898086383c P 42d1a4ef8e5b421a8c663ccf7a952cf5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:53:21.048921 19534 raft_consensus.cc:2229] T 7daa38aa4e564338a47e58898086383c P ececadece1c64419b1d4e8050c3c1fc5 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:53:21.049011 19533 raft_consensus.cc:2258] T 7daa38aa4e564338a47e58898086383c P 42d1a4ef8e5b421a8c663ccf7a952cf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:53:21.049022 19534 raft_consensus.cc:2258] T 7daa38aa4e564338a47e58898086383c P ececadece1c64419b1d4e8050c3c1fc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:53:21.049185 19535 tablet_replica.cc:330] T 7daa38aa4e564338a47e58898086383c P 96fa63cbf0c84edfaf66ae55042bd8df: stopping tablet replica
I20260619 16:53:21.049206 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 19106
I20260619 16:53:21.049292 19535 raft_consensus.cc:2229] T 7daa38aa4e564338a47e58898086383c P 96fa63cbf0c84edfaf66ae55042bd8df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:53:21.049386 19535 raft_consensus.cc:2258] T 7daa38aa4e564338a47e58898086383c P 96fa63cbf0c84edfaf66ae55042bd8df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:53:21.049549 19534 ts_tablet_manager.cc:1905] T 7daa38aa4e564338a47e58898086383c P ececadece1c64419b1d4e8050c3c1fc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:53:21.049819 19535 ts_tablet_manager.cc:1905] T 7daa38aa4e564338a47e58898086383c P 96fa63cbf0c84edfaf66ae55042bd8df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:53:21.051769 19534 ts_tablet_manager.cc:1918] T 7daa38aa4e564338a47e58898086383c P ececadece1c64419b1d4e8050c3c1fc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:53:21.051852 19534 log.cc:1196] T 7daa38aa4e564338a47e58898086383c P ececadece1c64419b1d4e8050c3c1fc5: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-2/wal/wals/7daa38aa4e564338a47e58898086383c
I20260619 16:53:21.052150 19534 ts_tablet_manager.cc:1939] T 7daa38aa4e564338a47e58898086383c P ececadece1c64419b1d4e8050c3c1fc5: Deleting consensus metadata
I20260619 16:53:21.052734 19030 catalog_manager.cc:4841] TS ececadece1c64419b1d4e8050c3c1fc5 (127.0.98.3:45465): tablet 7daa38aa4e564338a47e58898086383c (table test-table [id=9102cecbef0d44768239395dde14b83a]) successfully deleted
I20260619 16:53:21.053124 19535 ts_tablet_manager.cc:1918] T 7daa38aa4e564338a47e58898086383c P 96fa63cbf0c84edfaf66ae55042bd8df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:53:21.053202 19535 log.cc:1196] T 7daa38aa4e564338a47e58898086383c P 96fa63cbf0c84edfaf66ae55042bd8df: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781887741903642-392-0/minicluster-data/ts-1/wal/wals/7daa38aa4e564338a47e58898086383c
I20260619 16:53:21.053434 19535 ts_tablet_manager.cc:1939] T 7daa38aa4e564338a47e58898086383c P 96fa63cbf0c84edfaf66ae55042bd8df: Deleting consensus metadata
I20260619 16:53:21.054016 19033 catalog_manager.cc:4841] TS 96fa63cbf0c84edfaf66ae55042bd8df (127.0.98.2:38455): tablet 7daa38aa4e564338a47e58898086383c (table test-table [id=9102cecbef0d44768239395dde14b83a]) successfully deleted
W20260619 16:53:21.059312 19033 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43131 (error 108)
I20260619 16:53:21.059562 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 19237
I20260619 16:53:21.060331 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:21.059814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43131 (local address 127.0.0.1:58682)
0619 16:53:21.060204 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:21.060278 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354}
W20260619 16:53:21.060745 19033 catalog_manager.cc:4584] TS 42d1a4ef8e5b421a8c663ccf7a952cf5 (127.0.98.1:43131): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7daa38aa4e564338a47e58898086383c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43131: connect: Connection refused (error 111)
I20260619 16:53:21.064396 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 19368
I20260619 16:53:21.070097 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 19015
2026-06-19T16:53:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1011 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:53:21 dist-test-slave-jsw9 krb5kdc[19539](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:53:21 dist-test-slave-jsw9 krb5kdc[19539](info): set up 2 sockets
Jun 19 16:53:21 dist-test-slave-jsw9 krb5kdc[19539](info): commencing operation
krb5kdc: starting...
W20260619 16:53:22.034337 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:53:22 dist-test-slave-jsw9 krb5kdc[19539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:53:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:53:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:53:22.161510 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39453
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:53:22.252655 19552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:22.253032 19552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:22.253072 19552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:22.254626 19552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:53:22.254685 19552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:22.254707 19552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:22.254724 19552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:53:22.254739 19552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:53:22.256675 19552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:22.256994 19552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:22.257246 19552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:22.263370 19558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:22.263556 19560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:22.263391 19557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:22.263690 19552 server_base.cc:1034] running on GCE node
I20260619 16:53:22.264060 19552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:22.264429 19552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:22.265604 19552 hybrid_clock.cc:648] HybridClock initialized: now 1781888002265571 us; error 55 us; skew 500 ppm
Jun 19 16:53:22 dist-test-slave-jsw9 krb5kdc[19539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:22.268395 19552 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:53:22.268915 19552 webserver.cc:460] Webserver started at http://127.0.98.62:45283/ using document root <none> and password file <none>
I20260619 16:53:22.269121 19552 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:22.269164 19552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:22.269279 19552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:22.270512 19552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "1e164b7c4a944d2fbaf4630a0be05606"
format_stamp: "Formatted at 2026-06-19 16:53:22 on dist-test-slave-jsw9"
server_key: "7f5a98fcbd0ba49288a851f1532f0586"
server_key_iv: "f979427ddabec372e647a3fc61d8303d"
server_key_version: "encryptionkey@0"
I20260619 16:53:22.270839 19552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "1e164b7c4a944d2fbaf4630a0be05606"
format_stamp: "Formatted at 2026-06-19 16:53:22 on dist-test-slave-jsw9"
server_key: "7f5a98fcbd0ba49288a851f1532f0586"
server_key_iv: "f979427ddabec372e647a3fc61d8303d"
server_key_version: "encryptionkey@0"
I20260619 16:53:22.272210 19552 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 16:53:22.273098 19567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:22.273344 19552 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:53:22.273450 19552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "1e164b7c4a944d2fbaf4630a0be05606"
format_stamp: "Formatted at 2026-06-19 16:53:22 on dist-test-slave-jsw9"
server_key: "7f5a98fcbd0ba49288a851f1532f0586"
server_key_iv: "f979427ddabec372e647a3fc61d8303d"
server_key_version: "encryptionkey@0"
I20260619 16:53:22.273536 19552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:22.292137 19552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:22.294565 19552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:22.294710 19552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:22.299299 19552 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39453
I20260619 16:53:22.299365 19619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39453 every 8 connection(s)
I20260619 16:53:22.299737 19552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:53:22.300751 19620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:22.303478 19620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e164b7c4a944d2fbaf4630a0be05606: Bootstrap starting.
I20260619 16:53:22.304149 19620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e164b7c4a944d2fbaf4630a0be05606: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:22.304463 19620 log.cc:824] T 00000000000000000000000000000000 P 1e164b7c4a944d2fbaf4630a0be05606: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:22.305274 19620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e164b7c4a944d2fbaf4630a0be05606: No bootstrap required, opened a new log
I20260619 16:53:22.307163 19620 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e164b7c4a944d2fbaf4630a0be05606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e164b7c4a944d2fbaf4630a0be05606" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39453 } }
I20260619 16:53:22.307331 19620 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e164b7c4a944d2fbaf4630a0be05606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:22.307355 19620 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e164b7c4a944d2fbaf4630a0be05606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e164b7c4a944d2fbaf4630a0be05606, State: Initialized, Role: FOLLOWER
I20260619 16:53:22.307531 19620 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e164b7c4a944d2fbaf4630a0be05606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e164b7c4a944d2fbaf4630a0be05606" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39453 } }
I20260619 16:53:22.307626 19620 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e164b7c4a944d2fbaf4630a0be05606 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:53:22.307658 19620 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e164b7c4a944d2fbaf4630a0be05606 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:53:22.307703 19620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e164b7c4a944d2fbaf4630a0be05606 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:22.307906 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 19552
I20260619 16:53:22.308054 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:53:22.308266 392 external_mini_cluster.cc:1428] Setting key 5570b2d697218eb8a2827bdb79052fac
I20260619 16:53:22.308564 19620 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e164b7c4a944d2fbaf4630a0be05606 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e164b7c4a944d2fbaf4630a0be05606" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39453 } }
I20260619 16:53:22.308671 19620 leader_election.cc:302] T 00000000000000000000000000000000 P 1e164b7c4a944d2fbaf4630a0be05606 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e164b7c4a944d2fbaf4630a0be05606; no voters:
I20260619 16:53:22.308840 19620 leader_election.cc:288] T 00000000000000000000000000000000 P 1e164b7c4a944d2fbaf4630a0be05606 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:53:22.309015 19623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e164b7c4a944d2fbaf4630a0be05606 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:53:22.309137 19623 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e164b7c4a944d2fbaf4630a0be05606 [term 1 LEADER]: Becoming Leader. State: Replica: 1e164b7c4a944d2fbaf4630a0be05606, State: Running, Role: LEADER
I20260619 16:53:22.309154 19620 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e164b7c4a944d2fbaf4630a0be05606 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:53:22.309227 19623 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e164b7c4a944d2fbaf4630a0be05606 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e164b7c4a944d2fbaf4630a0be05606" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39453 } }
I20260619 16:53:22.309599 19623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e164b7c4a944d2fbaf4630a0be05606 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e164b7c4a944d2fbaf4630a0be05606. Latest consensus state: current_term: 1 leader_uuid: "1e164b7c4a944d2fbaf4630a0be05606" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e164b7c4a944d2fbaf4630a0be05606" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39453 } } }
I20260619 16:53:22.309700 19623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e164b7c4a944d2fbaf4630a0be05606 [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:22.309970 19633 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:53:22.310274 19627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e164b7c4a944d2fbaf4630a0be05606 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e164b7c4a944d2fbaf4630a0be05606" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e164b7c4a944d2fbaf4630a0be05606" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39453 } } }
I20260619 16:53:22.310520 19627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e164b7c4a944d2fbaf4630a0be05606 [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:22.310976 19633 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:53:22.313288 19633 catalog_manager.cc:1347] Generated new cluster ID: 8f3525e9c41348ac94b51934b3dc7e81
I20260619 16:53:22.313356 19633 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 16:53:22 dist-test-slave-jsw9 krb5kdc[19539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:53:22.321288 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:22.308828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53934 (local address 127.0.98.62:39453)
0619 16:53:22.309021 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:22.309033 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:22.309092 (+ 59us) server_negotiation.cc:407] Connection header received
0619 16:53:22.309249 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:22.309259 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:22.309336 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:22.309486 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:22.310342 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:22.312022 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:22.312920 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:22.313538 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:22.313664 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:22.317017 (+ 3353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:22.317023 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:22.317023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:22.317040 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:22.319226 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:22.319719 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:22.319722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:22.319723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:22.319857 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:22.320070 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:22.320071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:22.320072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:22.320425 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:22.320587 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:22.320722 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 16:53:22.320877 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":92,"threads_started":1}
I20260619 16:53:22.329954 19633 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:53:22.330996 19633 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:53:22.347291 19633 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e164b7c4a944d2fbaf4630a0be05606: Generated new TSK 0
I20260619 16:53:22.347575 19633 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:53:22.382571 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39453
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:53:22.475207 19646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:22.475498 19646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:22.475533 19646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:22.476928 19646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:22.476985 19646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:22.477016 19646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:53:22.478837 19646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:22.479105 19646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:22.479331 19646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:22.484838 19651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:22.484839 19654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:22.484843 19652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:22.485415 19646 server_base.cc:1034] running on GCE node
I20260619 16:53:22.485661 19646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:22.485941 19646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:22.487112 19646 hybrid_clock.cc:648] HybridClock initialized: now 1781888002487074 us; error 58 us; skew 500 ppm
Jun 19 16:53:22 dist-test-slave-jsw9 krb5kdc[19539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:22.490031 19646 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:53:22.490640 19646 webserver.cc:460] Webserver started at http://127.0.98.1:43981/ using document root <none> and password file <none>
I20260619 16:53:22.490864 19646 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:22.490913 19646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:22.491048 19646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:22.492282 19646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-0/data/instance:
uuid: "8381cc152c3c45e0af11e33948906458"
format_stamp: "Formatted at 2026-06-19 16:53:22 on dist-test-slave-jsw9"
server_key: "2171f325219f784ca75e2fd6eea6de0b"
server_key_iv: "5319d00e2c788e1a7e229d22e3f2ffb3"
server_key_version: "encryptionkey@0"
I20260619 16:53:22.492641 19646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8381cc152c3c45e0af11e33948906458"
format_stamp: "Formatted at 2026-06-19 16:53:22 on dist-test-slave-jsw9"
server_key: "2171f325219f784ca75e2fd6eea6de0b"
server_key_iv: "5319d00e2c788e1a7e229d22e3f2ffb3"
server_key_version: "encryptionkey@0"
I20260619 16:53:22.494081 19646 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 16:53:22.495011 19661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:22.495236 19646 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:53:22.495369 19646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-0/wal
uuid: "8381cc152c3c45e0af11e33948906458"
format_stamp: "Formatted at 2026-06-19 16:53:22 on dist-test-slave-jsw9"
server_key: "2171f325219f784ca75e2fd6eea6de0b"
server_key_iv: "5319d00e2c788e1a7e229d22e3f2ffb3"
server_key_version: "encryptionkey@0"
I20260619 16:53:22.495447 19646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:22.515717 19646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:22.518064 19646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:22.518258 19646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:22.518544 19646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:53:22.518906 19646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:53:22.518939 19646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:22.518985 19646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:53:22.519001 19646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:22.526410 19646 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33707
I20260619 16:53:22.526480 19774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33707 every 8 connection(s)
I20260619 16:53:22.526862 19646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:53:22.528924 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 19646
I20260619 16:53:22.529022 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:53:22.529263 392 external_mini_cluster.cc:1428] Setting key 0b5bd90f0bb552668d7405fcc48cf421
Jun 19 16:53:22 dist-test-slave-jsw9 krb5kdc[19539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:53:22.538894 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:22.527803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36781 (local address 127.0.98.62:39453)
0619 16:53:22.527869 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:22.527871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:22.528058 (+ 187us) server_negotiation.cc:407] Connection header received
0619 16:53:22.529179 (+ 1121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:22.529182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:22.529205 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:22.529290 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:22.530438 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:22.531206 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:22.532482 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:22.532790 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:22.532878 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:22.535660 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:22.535662 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:22.535663 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:22.535674 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:22.537268 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:22.537772 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:22.537773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:22.537773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:22.537842 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:22.538072 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:22.538073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:22.538073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:22.538268 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:22.538352 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:22.538671 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 16:53:22.538777 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 16:53:22.539022 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:22.527697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39453 (local address 127.0.98.1:36781)
0619 16:53:22.527889 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:22.527940 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:22.528971 (+ 1031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:22.529463 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:22.529469 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:22.529939 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:22.530310 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:22.530319 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:22.531354 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:22.531355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:22.532384 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:22.532387 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:22.532904 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:22.532905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:22.533058 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:22.533697 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:53:22.533707 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:22.535530 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:22.537405 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:22.537414 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:22.537418 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:22.537692 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:22.537906 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:22.537906 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:22.537907 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:22.538015 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:22.538348 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:22.538356 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:22.538522 (+ 166us) client_negotiation.cc:769] Sending connection context
0619 16:53:22.538628 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 16:53:22.538790 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":87,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 16:53:22.539621 19775 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39453
I20260619 16:53:22.539731 19775 heartbeater.cc:464] Registering TS with master...
I20260619 16:53:22.539939 19775 heartbeater.cc:511] Master 127.0.98.62:39453 requested a full tablet report, sending...
I20260619 16:53:22.540623 19584 ts_manager.cc:194] Registered new tserver with Master: 8381cc152c3c45e0af11e33948906458 (127.0.98.1:33707)
I20260619 16:53:22.541306 19584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36781
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:53:22.561079 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39453
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:53:22.652312 19780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:22.652573 19780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:22.652598 19780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:22.653813 19780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:22.653857 19780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:22.653884 19780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:53:22.655503 19780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:22.655747 19780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:22.655967 19780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:22.661813 19786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:22.661947 19780 server_base.cc:1034] running on GCE node
W20260619 16:53:22.661939 19785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:22.661868 19788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:22.662359 19780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:22.662636 19780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:22.663789 19780 hybrid_clock.cc:648] HybridClock initialized: now 1781888002663771 us; error 36 us; skew 500 ppm
Jun 19 16:53:22 dist-test-slave-jsw9 krb5kdc[19539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:22.667177 19780 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:53:22.667712 19780 webserver.cc:460] Webserver started at http://127.0.98.2:40887/ using document root <none> and password file <none>
I20260619 16:53:22.667922 19780 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:22.667968 19780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:22.668080 19780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:22.669289 19780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-1/data/instance:
uuid: "1e75d7f18b4b4162a76f3657f99119ba"
format_stamp: "Formatted at 2026-06-19 16:53:22 on dist-test-slave-jsw9"
server_key: "a477ff9c460a6e6bd0d860ccd7155738"
server_key_iv: "9f429ef89b3463af612d4068147d2870"
server_key_version: "encryptionkey@0"
I20260619 16:53:22.669660 19780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1e75d7f18b4b4162a76f3657f99119ba"
format_stamp: "Formatted at 2026-06-19 16:53:22 on dist-test-slave-jsw9"
server_key: "a477ff9c460a6e6bd0d860ccd7155738"
server_key_iv: "9f429ef89b3463af612d4068147d2870"
server_key_version: "encryptionkey@0"
I20260619 16:53:22.671334 19780 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 16:53:22.672319 19795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:22.672612 19780 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:53:22.672712 19780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-1/wal
uuid: "1e75d7f18b4b4162a76f3657f99119ba"
format_stamp: "Formatted at 2026-06-19 16:53:22 on dist-test-slave-jsw9"
server_key: "a477ff9c460a6e6bd0d860ccd7155738"
server_key_iv: "9f429ef89b3463af612d4068147d2870"
server_key_version: "encryptionkey@0"
I20260619 16:53:22.672799 19780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:22.684732 19780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:22.687282 19780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:22.687422 19780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:22.687693 19780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:53:22.688043 19780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:53:22.688078 19780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:22.688112 19780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:53:22.688135 19780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:22.694823 19780 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39521
I20260619 16:53:22.694868 19908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39521 every 8 connection(s)
I20260619 16:53:22.695235 19780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:53:22.697270 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 19780
I20260619 16:53:22.697386 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:53:22.697605 392 external_mini_cluster.cc:1428] Setting key 8e5dd5b66c204441faf24ae6fd3f7d12
Jun 19 16:53:22 dist-test-slave-jsw9 krb5kdc[19539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:53:22.706907 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:22.696073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54337 (local address 127.0.98.62:39453)
0619 16:53:22.696170 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:22.696176 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:22.696384 (+ 208us) server_negotiation.cc:407] Connection header received
0619 16:53:22.697502 (+ 1118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:22.697505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:22.697521 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:22.697601 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:22.698488 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:22.699218 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:22.700646 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:22.700857 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:22.700924 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:22.703641 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:22.703643 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:22.703644 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:22.703653 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:22.705227 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:22.705684 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:22.705685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:22.705686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:22.705755 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:22.706081 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:22.706082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:22.706082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:22.706280 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:22.706353 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:22.706679 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 16:53:22.706787 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 16:53:22.707031 19911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:22.696039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39453 (local address 127.0.98.2:54337)
0619 16:53:22.696213 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:22.696262 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:22.697317 (+ 1055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:22.697739 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:22.697743 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:22.698059 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:22.698361 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:22.698366 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:22.699319 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:22.699320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:22.700546 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:22.700549 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:22.700948 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:22.700949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:22.701109 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:22.701715 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:53:22.701724 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:22.703533 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:22.705335 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:22.705343 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:22.705345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:22.705607 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:22.705843 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:22.705844 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:22.705844 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:22.706000 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:22.706371 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:22.706378 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:22.706534 (+ 156us) client_negotiation.cc:769] Sending connection context
0619 16:53:22.706633 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 16:53:22.706790 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 16:53:22.707578 19909 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39453
I20260619 16:53:22.707686 19909 heartbeater.cc:464] Registering TS with master...
I20260619 16:53:22.707887 19909 heartbeater.cc:511] Master 127.0.98.62:39453 requested a full tablet report, sending...
I20260619 16:53:22.708465 19584 ts_manager.cc:194] Registered new tserver with Master: 1e75d7f18b4b4162a76f3657f99119ba (127.0.98.2:39521)
I20260619 16:53:22.708915 19584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54337
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:53:22.730428 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39453
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:53:22.822072 19914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:22.822414 19914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:22.822456 19914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:22.823880 19914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:22.823937 19914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:22.823980 19914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:53:22.825723 19914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:22.826005 19914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:22.826287 19914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:53:22.831871 19914 server_base.cc:1034] running on GCE node
W20260619 16:53:22.831875 19920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:22.831913 19919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:22.832056 19922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:22.832341 19914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:22.832623 19914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:22.833807 19914 hybrid_clock.cc:648] HybridClock initialized: now 1781888002833793 us; error 33 us; skew 500 ppm
Jun 19 16:53:22 dist-test-slave-jsw9 krb5kdc[19539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:22.836763 19914 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:53:22.837319 19914 webserver.cc:460] Webserver started at http://127.0.98.3:43223/ using document root <none> and password file <none>
I20260619 16:53:22.837576 19914 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:22.837628 19914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:22.837754 19914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:22.839035 19914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-2/data/instance:
uuid: "8e8a11fb24224c28851c4eaae69c9aeb"
format_stamp: "Formatted at 2026-06-19 16:53:22 on dist-test-slave-jsw9"
server_key: "573a1df4bdaffd26f4bbf77cfb155e2a"
server_key_iv: "89338463d3617d2371d32d9cb50b66be"
server_key_version: "encryptionkey@0"
I20260619 16:53:22.839402 19914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8e8a11fb24224c28851c4eaae69c9aeb"
format_stamp: "Formatted at 2026-06-19 16:53:22 on dist-test-slave-jsw9"
server_key: "573a1df4bdaffd26f4bbf77cfb155e2a"
server_key_iv: "89338463d3617d2371d32d9cb50b66be"
server_key_version: "encryptionkey@0"
I20260619 16:53:22.840817 19914 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 16:53:22.841586 19929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:22.841741 19914 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 16:53:22.841850 19914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-2/wal
uuid: "8e8a11fb24224c28851c4eaae69c9aeb"
format_stamp: "Formatted at 2026-06-19 16:53:22 on dist-test-slave-jsw9"
server_key: "573a1df4bdaffd26f4bbf77cfb155e2a"
server_key_iv: "89338463d3617d2371d32d9cb50b66be"
server_key_version: "encryptionkey@0"
I20260619 16:53:22.841919 19914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:22.869807 19914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:22.872769 19914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:22.872910 19914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:22.873136 19914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:53:22.873502 19914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:53:22.873538 19914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:22.873575 19914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:53:22.873598 19914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:22.880828 19914 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36705
I20260619 16:53:22.880997 20042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36705 every 8 connection(s)
I20260619 16:53:22.881302 19914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:53:22.886816 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 19914
I20260619 16:53:22.886930 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:53:22.887125 392 external_mini_cluster.cc:1428] Setting key 7d1037de9785d70cde91dd56d13f7400
Jun 19 16:53:22 dist-test-slave-jsw9 krb5kdc[19539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:53:22.892444 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:22.882321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58121 (local address 127.0.98.62:39453)
0619 16:53:22.882387 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:22.882390 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:22.882687 (+ 297us) server_negotiation.cc:407] Connection header received
0619 16:53:22.883797 (+ 1110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:22.883799 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:22.883816 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:22.883895 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:22.884711 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:22.885243 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:22.886411 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:22.886627 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:22.886697 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:22.889498 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:22.889499 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:22.889500 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:22.889509 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:22.890965 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:22.891401 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:22.891402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:22.891403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:22.891465 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:22.891705 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:22.891706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:22.891707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:22.891856 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:22.891932 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:22.892229 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 16:53:22.892328 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 16:53:22.892642 20045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:22.882259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39453 (local address 127.0.98.3:58121)
0619 16:53:22.882464 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:22.882519 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:22.883636 (+ 1117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:22.884018 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:22.884023 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:22.884354 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:22.884602 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:22.884606 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:22.885335 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:22.885336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:22.886308 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:22.886311 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:22.886736 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:22.886738 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:22.886880 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:22.887618 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:53:22.887629 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:22.889396 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:22.891050 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:22.891061 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:22.891065 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:22.891323 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:22.891520 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:22.891520 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:22.891521 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:22.891647 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:22.891927 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:22.891934 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:22.892095 (+ 161us) client_negotiation.cc:769] Sending connection context
0619 16:53:22.892195 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 16:53:22.892387 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":98,"threads_started":1}
I20260619 16:53:22.893239 20043 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39453
I20260619 16:53:22.893352 20043 heartbeater.cc:464] Registering TS with master...
I20260619 16:53:22.893538 20043 heartbeater.cc:511] Master 127.0.98.62:39453 requested a full tablet report, sending...
I20260619 16:53:22.894089 19584 ts_manager.cc:194] Registered new tserver with Master: 8e8a11fb24224c28851c4eaae69c9aeb (127.0.98.3:36705)
I20260619 16:53:22.894625 19584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58121
I20260619 16:53:22.899689 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:53:22.908144 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:22.901209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53938 (local address 127.0.98.62:39453)
0619 16:53:22.901296 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:22.901298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:22.901356 (+ 58us) server_negotiation.cc:407] Connection header received
0619 16:53:22.901416 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:22.901417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:22.901426 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:22.901482 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:22.902057 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:22.902635 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:22.903473 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:22.903699 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:22.903769 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:22.905132 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:22.905133 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:22.905134 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:22.905141 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:22.906647 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:22.907054 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:22.907055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:22.907056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:22.907120 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:22.907478 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:22.907479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:22.907479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:22.907683 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:22.907777 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:22.907960 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 16:53:22.908042 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:53:22.910303 19584 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53938:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:53:22.912364 19584 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:53:22.920573 20057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:22.915552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36705 (local address 127.0.0.1:41528)
0619 16:53:22.916238 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:22.916248 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:22.916303 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:22.916521 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:22.916522 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:22.916524 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:22.916572 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:22.916573 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:22.917805 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:22.917806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:22.918786 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:22.918790 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:22.920054 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:22.920056 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:22.920259 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:22.920286 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 16:53:22.920376 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 16:53:22.920460 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"mutex_wait_us":3,"thread_start_us":74,"threads_started":1}
I20260619 16:53:22.920606 20059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:22.915705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40804 (local address 127.0.98.2:39521)
0619 16:53:22.915999 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:22.916003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:22.916012 (+ 9us) server_negotiation.cc:407] Connection header received
0619 16:53:22.916121 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:22.916124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:22.916141 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:22.916213 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:22.916421 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:22.917293 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:22.919041 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:22.920092 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:22.920158 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:22.920213 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:22.920447 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 16:53:22.920507 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":127,"threads_started":1}
I20260619 16:53:22.920653 20056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:22.915445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52940 (local address 127.0.98.1:33707)
0619 16:53:22.915911 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:22.915914 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:22.915926 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:53:22.915991 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:22.915994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:22.916012 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:22.916111 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:22.916385 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:22.917570 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:22.919097 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:22.919885 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:22.920040 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:22.920245 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:22.920512 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 16:53:22.920572 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":107,"threads_started":1}
I20260619 16:53:22.920936 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:22.915299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33707 (local address 127.0.0.1:52940)
0619 16:53:22.915699 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:22.915757 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:22.915840 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:22.916117 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:22.916120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:22.916123 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:22.916246 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:22.916249 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:22.917687 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:22.917688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:22.918976 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:22.918978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:22.920302 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:22.920302 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:22.920401 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:22.920405 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:53:22.920765 (+ 360us) client_negotiation.cc:240] Negotiation successful
0619 16:53:22.920815 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":57,"thread_start_us":68,"threads_started":1}
I20260619 16:53:22.921012 20058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:22.915702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41528 (local address 127.0.98.3:36705)
0619 16:53:22.915905 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:22.915912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:22.916361 (+ 449us) server_negotiation.cc:407] Connection header received
0619 16:53:22.916413 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:22.916415 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:22.916434 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:22.916514 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:22.916629 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:22.917707 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:22.918881 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:22.919886 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:22.919976 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:22.920053 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:22.920857 (+ 804us) server_negotiation.cc:299] Negotiation successful
0619 16:53:22.920919 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":113,"threads_started":1}
I20260619 16:53:22.921389 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:22.915301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39521 (local address 127.0.0.1:40804)
0619 16:53:22.915960 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:22.915970 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:22.916022 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:22.916301 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:22.916301 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:22.916304 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:22.916349 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:22.916351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:22.917407 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:22.917409 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:22.918923 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:22.918927 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:22.920163 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:22.920164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:22.920269 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:22.920284 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 16:53:22.920377 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 16:53:22.921326 (+ 949us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":878,"thread_start_us":122,"threads_started":1}
I20260619 16:53:22.921835 19709 tablet_service.cc:1467] Processing CreateTablet for tablet 28b621edd6a6424886a80593b9e4041a (DEFAULT_TABLE table=test-table [id=4b34cb0f0bfc4dcb945d8874b1db62f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:53:22.921833 19977 tablet_service.cc:1467] Processing CreateTablet for tablet 28b621edd6a6424886a80593b9e4041a (DEFAULT_TABLE table=test-table [id=4b34cb0f0bfc4dcb945d8874b1db62f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:53:22.922042 19843 tablet_service.cc:1467] Processing CreateTablet for tablet 28b621edd6a6424886a80593b9e4041a (DEFAULT_TABLE table=test-table [id=4b34cb0f0bfc4dcb945d8874b1db62f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:53:22.922318 19709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28b621edd6a6424886a80593b9e4041a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:22.922318 19977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28b621edd6a6424886a80593b9e4041a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:22.922436 19843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28b621edd6a6424886a80593b9e4041a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:22.925587 20060 tablet_bootstrap.cc:492] T 28b621edd6a6424886a80593b9e4041a P 1e75d7f18b4b4162a76f3657f99119ba: Bootstrap starting.
I20260619 16:53:22.925948 20061 tablet_bootstrap.cc:492] T 28b621edd6a6424886a80593b9e4041a P 8e8a11fb24224c28851c4eaae69c9aeb: Bootstrap starting.
I20260619 16:53:22.926591 20060 tablet_bootstrap.cc:654] T 28b621edd6a6424886a80593b9e4041a P 1e75d7f18b4b4162a76f3657f99119ba: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:22.926638 20061 tablet_bootstrap.cc:654] T 28b621edd6a6424886a80593b9e4041a P 8e8a11fb24224c28851c4eaae69c9aeb: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:22.927265 20061 log.cc:824] T 28b621edd6a6424886a80593b9e4041a P 8e8a11fb24224c28851c4eaae69c9aeb: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:22.927266 20060 log.cc:824] T 28b621edd6a6424886a80593b9e4041a P 1e75d7f18b4b4162a76f3657f99119ba: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:22.927656 20062 tablet_bootstrap.cc:492] T 28b621edd6a6424886a80593b9e4041a P 8381cc152c3c45e0af11e33948906458: Bootstrap starting.
I20260619 16:53:22.928194 20061 tablet_bootstrap.cc:492] T 28b621edd6a6424886a80593b9e4041a P 8e8a11fb24224c28851c4eaae69c9aeb: No bootstrap required, opened a new log
I20260619 16:53:22.928216 20060 tablet_bootstrap.cc:492] T 28b621edd6a6424886a80593b9e4041a P 1e75d7f18b4b4162a76f3657f99119ba: No bootstrap required, opened a new log
I20260619 16:53:22.928274 20060 ts_tablet_manager.cc:1397] T 28b621edd6a6424886a80593b9e4041a P 1e75d7f18b4b4162a76f3657f99119ba: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:53:22.928274 20061 ts_tablet_manager.cc:1397] T 28b621edd6a6424886a80593b9e4041a P 8e8a11fb24224c28851c4eaae69c9aeb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:53:22.928365 20062 tablet_bootstrap.cc:654] T 28b621edd6a6424886a80593b9e4041a P 8381cc152c3c45e0af11e33948906458: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:22.929018 20062 log.cc:824] T 28b621edd6a6424886a80593b9e4041a P 8381cc152c3c45e0af11e33948906458: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:22.929931 20062 tablet_bootstrap.cc:492] T 28b621edd6a6424886a80593b9e4041a P 8381cc152c3c45e0af11e33948906458: No bootstrap required, opened a new log
I20260619 16:53:22.930006 20062 ts_tablet_manager.cc:1397] T 28b621edd6a6424886a80593b9e4041a P 8381cc152c3c45e0af11e33948906458: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:53:22.930341 20060 raft_consensus.cc:348] T 28b621edd6a6424886a80593b9e4041a P 1e75d7f18b4b4162a76f3657f99119ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e75d7f18b4b4162a76f3657f99119ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39521 } } peers { permanent_uuid: "8e8a11fb24224c28851c4eaae69c9aeb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "8381cc152c3c45e0af11e33948906458" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33707 } }
I20260619 16:53:22.930514 20060 raft_consensus.cc:374] T 28b621edd6a6424886a80593b9e4041a P 1e75d7f18b4b4162a76f3657f99119ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:22.930549 20060 raft_consensus.cc:729] T 28b621edd6a6424886a80593b9e4041a P 1e75d7f18b4b4162a76f3657f99119ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e75d7f18b4b4162a76f3657f99119ba, State: Initialized, Role: FOLLOWER
I20260619 16:53:22.930562 20061 raft_consensus.cc:348] T 28b621edd6a6424886a80593b9e4041a P 8e8a11fb24224c28851c4eaae69c9aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e75d7f18b4b4162a76f3657f99119ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39521 } } peers { permanent_uuid: "8e8a11fb24224c28851c4eaae69c9aeb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "8381cc152c3c45e0af11e33948906458" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33707 } }
I20260619 16:53:22.930737 20061 raft_consensus.cc:374] T 28b621edd6a6424886a80593b9e4041a P 8e8a11fb24224c28851c4eaae69c9aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:22.930702 20060 consensus_queue.cc:260] T 28b621edd6a6424886a80593b9e4041a P 1e75d7f18b4b4162a76f3657f99119ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e75d7f18b4b4162a76f3657f99119ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39521 } } peers { permanent_uuid: "8e8a11fb24224c28851c4eaae69c9aeb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "8381cc152c3c45e0af11e33948906458" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33707 } }
I20260619 16:53:22.930775 20061 raft_consensus.cc:729] T 28b621edd6a6424886a80593b9e4041a P 8e8a11fb24224c28851c4eaae69c9aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e8a11fb24224c28851c4eaae69c9aeb, State: Initialized, Role: FOLLOWER
I20260619 16:53:22.930924 20061 consensus_queue.cc:260] T 28b621edd6a6424886a80593b9e4041a P 8e8a11fb24224c28851c4eaae69c9aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e75d7f18b4b4162a76f3657f99119ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39521 } } peers { permanent_uuid: "8e8a11fb24224c28851c4eaae69c9aeb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "8381cc152c3c45e0af11e33948906458" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33707 } }
I20260619 16:53:22.931017 19909 heartbeater.cc:503] Master 127.0.98.62:39453 was elected leader, sending a full tablet report...
I20260619 16:53:22.931021 20060 ts_tablet_manager.cc:1428] T 28b621edd6a6424886a80593b9e4041a P 1e75d7f18b4b4162a76f3657f99119ba: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260619 16:53:22.931218 20061 ts_tablet_manager.cc:1428] T 28b621edd6a6424886a80593b9e4041a P 8e8a11fb24224c28851c4eaae69c9aeb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 16:53:22.931341 20043 heartbeater.cc:503] Master 127.0.98.62:39453 was elected leader, sending a full tablet report...
I20260619 16:53:22.932670 20062 raft_consensus.cc:348] T 28b621edd6a6424886a80593b9e4041a P 8381cc152c3c45e0af11e33948906458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e75d7f18b4b4162a76f3657f99119ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39521 } } peers { permanent_uuid: "8e8a11fb24224c28851c4eaae69c9aeb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "8381cc152c3c45e0af11e33948906458" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33707 } }
I20260619 16:53:22.932821 20062 raft_consensus.cc:374] T 28b621edd6a6424886a80593b9e4041a P 8381cc152c3c45e0af11e33948906458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:22.932843 20062 raft_consensus.cc:729] T 28b621edd6a6424886a80593b9e4041a P 8381cc152c3c45e0af11e33948906458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8381cc152c3c45e0af11e33948906458, State: Initialized, Role: FOLLOWER
I20260619 16:53:22.932953 20062 consensus_queue.cc:260] T 28b621edd6a6424886a80593b9e4041a P 8381cc152c3c45e0af11e33948906458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e75d7f18b4b4162a76f3657f99119ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39521 } } peers { permanent_uuid: "8e8a11fb24224c28851c4eaae69c9aeb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "8381cc152c3c45e0af11e33948906458" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33707 } }
I20260619 16:53:22.933219 20062 ts_tablet_manager.cc:1428] T 28b621edd6a6424886a80593b9e4041a P 8381cc152c3c45e0af11e33948906458: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 16:53:22.933235 19775 heartbeater.cc:503] Master 127.0.98.62:39453 was elected leader, sending a full tablet report...
W20260619 16:53:22.945876 19910 tablet.cc:2369] T 28b621edd6a6424886a80593b9e4041a P 1e75d7f18b4b4162a76f3657f99119ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:53:22.958416 20068 raft_consensus.cc:482] T 28b621edd6a6424886a80593b9e4041a P 8381cc152c3c45e0af11e33948906458 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:53:22.958688 20068 raft_consensus.cc:504] T 28b621edd6a6424886a80593b9e4041a P 8381cc152c3c45e0af11e33948906458 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e75d7f18b4b4162a76f3657f99119ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39521 } } peers { permanent_uuid: "8e8a11fb24224c28851c4eaae69c9aeb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "8381cc152c3c45e0af11e33948906458" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33707 } }
I20260619 16:53:22.959172 20068 leader_election.cc:288] T 28b621edd6a6424886a80593b9e4041a P 8381cc152c3c45e0af11e33948906458 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e75d7f18b4b4162a76f3657f99119ba (127.0.98.2:39521), 8e8a11fb24224c28851c4eaae69c9aeb (127.0.98.3:36705)
I20260619 16:53:22.962239 20059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:22.959543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36459 (local address 127.0.98.2:39521)
0619 16:53:22.959622 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:22.959624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:22.959637 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:53:22.959671 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:22.959672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:22.959688 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:22.959750 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:22.959937 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:22.960464 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:22.961465 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:22.961874 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:22.961936 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:22.961950 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:22.962093 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 16:53:22.962136 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 16:53:22.962239 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:22.959455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39521 (local address 127.0.98.1:36459)
0619 16:53:22.959523 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:22.959539 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:22.959610 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:22.959760 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:22.959761 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:22.959763 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:22.959847 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:22.959849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:22.960546 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:22.960548 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:22.961366 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:22.961368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:22.961951 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:22.961952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:22.962027 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:22.962030 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:53:22.962077 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 16:53:22.962126 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 16:53:22.962334 20058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:22.959654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47535 (local address 127.0.98.3:36705)
0619 16:53:22.959717 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:22.959719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:22.959863 (+ 144us) server_negotiation.cc:407] Connection header received
0619 16:53:22.959914 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:22.959916 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:22.959924 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:22.959968 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:22.960124 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:22.960607 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:22.961564 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:22.961962 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:22.962045 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:22.962058 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:22.962206 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 16:53:22.962256 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 16:53:22.962366 20069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:22.959567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36705 (local address 127.0.98.1:47535)
0619 16:53:22.959794 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:22.959803 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:22.959856 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:22.959983 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:22.959984 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:22.959986 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:22.960064 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:22.960065 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:22.960686 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:22.960687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:22.961474 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:22.961476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:22.962045 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:22.962046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:22.962123 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:22.962126 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:53:22.962173 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 16:53:22.962242 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":156,"threads_started":1}
I20260619 16:53:22.962693 19997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28b621edd6a6424886a80593b9e4041a" candidate_uuid: "8381cc152c3c45e0af11e33948906458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e8a11fb24224c28851c4eaae69c9aeb" is_pre_election: true
I20260619 16:53:22.962687 19863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28b621edd6a6424886a80593b9e4041a" candidate_uuid: "8381cc152c3c45e0af11e33948906458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e75d7f18b4b4162a76f3657f99119ba" is_pre_election: true
I20260619 16:53:22.962867 19997 raft_consensus.cc:2454] T 28b621edd6a6424886a80593b9e4041a P 8e8a11fb24224c28851c4eaae69c9aeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8381cc152c3c45e0af11e33948906458 in term 0.
I20260619 16:53:22.962867 19863 raft_consensus.cc:2454] T 28b621edd6a6424886a80593b9e4041a P 1e75d7f18b4b4162a76f3657f99119ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8381cc152c3c45e0af11e33948906458 in term 0.
I20260619 16:53:22.963101 19663 leader_election.cc:302] T 28b621edd6a6424886a80593b9e4041a P 8381cc152c3c45e0af11e33948906458 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e75d7f18b4b4162a76f3657f99119ba, 8381cc152c3c45e0af11e33948906458; no voters:
I20260619 16:53:22.963246 20068 raft_consensus.cc:2781] T 28b621edd6a6424886a80593b9e4041a P 8381cc152c3c45e0af11e33948906458 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:53:22.963340 20068 raft_consensus.cc:482] T 28b621edd6a6424886a80593b9e4041a P 8381cc152c3c45e0af11e33948906458 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:53:22.963369 20068 raft_consensus.cc:3037] T 28b621edd6a6424886a80593b9e4041a P 8381cc152c3c45e0af11e33948906458 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:22.964219 20068 raft_consensus.cc:504] T 28b621edd6a6424886a80593b9e4041a P 8381cc152c3c45e0af11e33948906458 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e75d7f18b4b4162a76f3657f99119ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39521 } } peers { permanent_uuid: "8e8a11fb24224c28851c4eaae69c9aeb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "8381cc152c3c45e0af11e33948906458" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33707 } }
I20260619 16:53:22.964368 20068 leader_election.cc:288] T 28b621edd6a6424886a80593b9e4041a P 8381cc152c3c45e0af11e33948906458 [CANDIDATE]: Term 1 election: Requested vote from peers 1e75d7f18b4b4162a76f3657f99119ba (127.0.98.2:39521), 8e8a11fb24224c28851c4eaae69c9aeb (127.0.98.3:36705)
I20260619 16:53:22.964630 19997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28b621edd6a6424886a80593b9e4041a" candidate_uuid: "8381cc152c3c45e0af11e33948906458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e8a11fb24224c28851c4eaae69c9aeb"
I20260619 16:53:22.964661 19863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28b621edd6a6424886a80593b9e4041a" candidate_uuid: "8381cc152c3c45e0af11e33948906458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e75d7f18b4b4162a76f3657f99119ba"
I20260619 16:53:22.964728 19997 raft_consensus.cc:3037] T 28b621edd6a6424886a80593b9e4041a P 8e8a11fb24224c28851c4eaae69c9aeb [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:22.964771 19863 raft_consensus.cc:3037] T 28b621edd6a6424886a80593b9e4041a P 1e75d7f18b4b4162a76f3657f99119ba [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:22.965653 19997 raft_consensus.cc:2454] T 28b621edd6a6424886a80593b9e4041a P 8e8a11fb24224c28851c4eaae69c9aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8381cc152c3c45e0af11e33948906458 in term 1.
I20260619 16:53:22.965685 19863 raft_consensus.cc:2454] T 28b621edd6a6424886a80593b9e4041a P 1e75d7f18b4b4162a76f3657f99119ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8381cc152c3c45e0af11e33948906458 in term 1.
I20260619 16:53:22.965849 19663 leader_election.cc:302] T 28b621edd6a6424886a80593b9e4041a P 8381cc152c3c45e0af11e33948906458 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8381cc152c3c45e0af11e33948906458, 8e8a11fb24224c28851c4eaae69c9aeb; no voters:
I20260619 16:53:22.965958 20068 raft_consensus.cc:2781] T 28b621edd6a6424886a80593b9e4041a P 8381cc152c3c45e0af11e33948906458 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:53:22.966159 20068 raft_consensus.cc:686] T 28b621edd6a6424886a80593b9e4041a P 8381cc152c3c45e0af11e33948906458 [term 1 LEADER]: Becoming Leader. State: Replica: 8381cc152c3c45e0af11e33948906458, State: Running, Role: LEADER
I20260619 16:53:22.966324 20068 consensus_queue.cc:237] T 28b621edd6a6424886a80593b9e4041a P 8381cc152c3c45e0af11e33948906458 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e75d7f18b4b4162a76f3657f99119ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39521 } } peers { permanent_uuid: "8e8a11fb24224c28851c4eaae69c9aeb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } } peers { permanent_uuid: "8381cc152c3c45e0af11e33948906458" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33707 } }
I20260619 16:53:22.967356 19583 catalog_manager.cc:5495] T 28b621edd6a6424886a80593b9e4041a P 8381cc152c3c45e0af11e33948906458 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8381cc152c3c45e0af11e33948906458 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8381cc152c3c45e0af11e33948906458" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e75d7f18b4b4162a76f3657f99119ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e8a11fb24224c28851c4eaae69c9aeb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8381cc152c3c45e0af11e33948906458" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33707 } health_report { overall_health: HEALTHY } } }
I20260619 16:53:22.982518 20056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:22.979610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52954 (local address 127.0.98.1:33707)
0619 16:53:22.979680 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:22.979682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:22.979689 (+ 7us) server_negotiation.cc:407] Connection header received
0619 16:53:22.979730 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:22.979731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:22.979755 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:22.979819 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:53:22.979982 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:22.980491 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:22.981497 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:22.981782 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:22.981884 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:22.982061 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:53:22.982239 (+ 178us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:53:22.982322 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:22.982363 (+ 41us) server_negotiation.cc:299] Negotiation successful
0619 16:53:22.982391 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 16:53:22.984817 19863 raft_consensus.cc:1261] T 28b621edd6a6424886a80593b9e4041a P 1e75d7f18b4b4162a76f3657f99119ba [term 1 FOLLOWER]: Refusing update from remote peer 8381cc152c3c45e0af11e33948906458: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:53:22.984833 19997 raft_consensus.cc:1261] T 28b621edd6a6424886a80593b9e4041a P 8e8a11fb24224c28851c4eaae69c9aeb [term 1 FOLLOWER]: Refusing update from remote peer 8381cc152c3c45e0af11e33948906458: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:53:22.985209 20070 consensus_queue.cc:1035] T 28b621edd6a6424886a80593b9e4041a P 8381cc152c3c45e0af11e33948906458 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e8a11fb24224c28851c4eaae69c9aeb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:53:22.985390 20068 consensus_queue.cc:1035] T 28b621edd6a6424886a80593b9e4041a P 8381cc152c3c45e0af11e33948906458 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e75d7f18b4b4162a76f3657f99119ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:53:22.988582 20073 mvcc.cc:205] Tried to move back new op lower bound from 7298613260222230528 to 7298613260150669312. Current Snapshot: MvccSnapshot[applied={T|T < 7298613260222230528}]
I20260619 16:53:22.988901 20075 mvcc.cc:205] Tried to move back new op lower bound from 7298613260222230528 to 7298613260150669312. Current Snapshot: MvccSnapshot[applied={T|T < 7298613260222230528}]
I20260619 16:53:22.991683 19583 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:53:22.991799 19583 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:53:22.992758 19583 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e164b7c4a944d2fbaf4630a0be05606: Sending DeleteTablet for 3 replicas of tablet 28b621edd6a6424886a80593b9e4041a
I20260619 16:53:22.993158 19843 tablet_service.cc:1514] Processing DeleteTablet for tablet 28b621edd6a6424886a80593b9e4041a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:53:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40804
I20260619 16:53:22.993173 19977 tablet_service.cc:1514] Processing DeleteTablet for tablet 28b621edd6a6424886a80593b9e4041a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:53:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41528
I20260619 16:53:22.993443 19709 tablet_service.cc:1514] Processing DeleteTablet for tablet 28b621edd6a6424886a80593b9e4041a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:53:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52940
I20260619 16:53:22.993667 20085 tablet_replica.cc:330] T 28b621edd6a6424886a80593b9e4041a P 8381cc152c3c45e0af11e33948906458: stopping tablet replica
I20260619 16:53:22.993796 20084 tablet_replica.cc:330] T 28b621edd6a6424886a80593b9e4041a P 1e75d7f18b4b4162a76f3657f99119ba: stopping tablet replica
I20260619 16:53:22.993952 20085 raft_consensus.cc:2229] T 28b621edd6a6424886a80593b9e4041a P 8381cc152c3c45e0af11e33948906458 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:53:22.993971 20084 raft_consensus.cc:2229] T 28b621edd6a6424886a80593b9e4041a P 1e75d7f18b4b4162a76f3657f99119ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:53:22.994167 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 19646
I20260619 16:53:22.994982 20084 raft_consensus.cc:2258] T 28b621edd6a6424886a80593b9e4041a P 1e75d7f18b4b4162a76f3657f99119ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:53:22.995502 20084 ts_tablet_manager.cc:1905] T 28b621edd6a6424886a80593b9e4041a P 1e75d7f18b4b4162a76f3657f99119ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:53:22.997056 20083 tablet_replica.cc:330] T 28b621edd6a6424886a80593b9e4041a P 8e8a11fb24224c28851c4eaae69c9aeb: stopping tablet replica
I20260619 16:53:22.997215 20083 raft_consensus.cc:2229] T 28b621edd6a6424886a80593b9e4041a P 8e8a11fb24224c28851c4eaae69c9aeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:53:22.997390 20083 raft_consensus.cc:2258] T 28b621edd6a6424886a80593b9e4041a P 8e8a11fb24224c28851c4eaae69c9aeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:53:22.997390 20084 ts_tablet_manager.cc:1918] T 28b621edd6a6424886a80593b9e4041a P 1e75d7f18b4b4162a76f3657f99119ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:53:22.997460 20084 log.cc:1196] T 28b621edd6a6424886a80593b9e4041a P 1e75d7f18b4b4162a76f3657f99119ba: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-1/wal/wals/28b621edd6a6424886a80593b9e4041a
I20260619 16:53:22.997761 20084 ts_tablet_manager.cc:1939] T 28b621edd6a6424886a80593b9e4041a P 1e75d7f18b4b4162a76f3657f99119ba: Deleting consensus metadata
I20260619 16:53:22.997978 20083 ts_tablet_manager.cc:1905] T 28b621edd6a6424886a80593b9e4041a P 8e8a11fb24224c28851c4eaae69c9aeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:53:22.998270 19569 catalog_manager.cc:4841] TS 1e75d7f18b4b4162a76f3657f99119ba (127.0.98.2:39521): tablet 28b621edd6a6424886a80593b9e4041a (table test-table [id=4b34cb0f0bfc4dcb945d8874b1db62f1]) successfully deleted
I20260619 16:53:22.999502 20083 ts_tablet_manager.cc:1918] T 28b621edd6a6424886a80593b9e4041a P 8e8a11fb24224c28851c4eaae69c9aeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:53:22.999567 20083 log.cc:1196] T 28b621edd6a6424886a80593b9e4041a P 8e8a11fb24224c28851c4eaae69c9aeb: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781887741903642-392-0/minicluster-data/ts-2/wal/wals/28b621edd6a6424886a80593b9e4041a
I20260619 16:53:22.999794 20083 ts_tablet_manager.cc:1939] T 28b621edd6a6424886a80593b9e4041a P 8e8a11fb24224c28851c4eaae69c9aeb: Deleting consensus metadata
I20260619 16:53:23.000284 19569 catalog_manager.cc:4841] TS 8e8a11fb24224c28851c4eaae69c9aeb (127.0.98.3:36705): tablet 28b621edd6a6424886a80593b9e4041a (table test-table [id=4b34cb0f0bfc4dcb945d8874b1db62f1]) successfully deleted
W20260619 16:53:23.000350 19568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33707 (error 108)
I20260619 16:53:23.000677 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 19780
I20260619 16:53:23.001308 20057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:23.000828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33707 (local address 127.0.0.1:52966)
0619 16:53:23.001225 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:23.001270 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373}
W20260619 16:53:23.001452 19568 catalog_manager.cc:4584] TS 8381cc152c3c45e0af11e33948906458 (127.0.98.1:33707): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 28b621edd6a6424886a80593b9e4041a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33707: connect: Connection refused (error 111)
I20260619 16:53:23.006103 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 19914
I20260619 16:53:23.011000 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 19552
2026-06-19T16:53:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1941 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T16:53:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:53:23Z Disabled control of system clock
I20260619 16:53:23.040628 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39947
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:53:23.137746 20090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:23.138060 20090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:23.138087 20090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:23.139570 20090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:53:23.139623 20090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:23.139635 20090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:23.139647 20090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:53:23.139658 20090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:53:23.141168 20090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39947
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:23.141443 20090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:23.141755 20090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:23.147634 20098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:23.147600 20095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:23.147714 20090 server_base.cc:1034] running on GCE node
W20260619 16:53:23.147591 20096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:23.148154 20090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:23.148526 20090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:23.149688 20090 hybrid_clock.cc:648] HybridClock initialized: now 1781888003149669 us; error 37 us; skew 500 ppm
I20260619 16:53:23.151248 20090 webserver.cc:460] Webserver started at http://127.0.98.62:34709/ using document root <none> and password file <none>
I20260619 16:53:23.151561 20090 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:23.151621 20090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:23.151765 20090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:23.153060 20090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "50b8a91a2dff4a86b84ecd53c1df89ed"
format_stamp: "Formatted at 2026-06-19 16:53:23 on dist-test-slave-jsw9"
server_key: "15b6badb5306e8ed9712de48b820246e"
server_key_iv: "ec957ffd8ffbe5394e7037fca80c2b7d"
server_key_version: "encryptionkey@0"
I20260619 16:53:23.153469 20090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "50b8a91a2dff4a86b84ecd53c1df89ed"
format_stamp: "Formatted at 2026-06-19 16:53:23 on dist-test-slave-jsw9"
server_key: "15b6badb5306e8ed9712de48b820246e"
server_key_iv: "ec957ffd8ffbe5394e7037fca80c2b7d"
server_key_version: "encryptionkey@0"
I20260619 16:53:23.154995 20090 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 16:53:23.155895 20104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:23.156105 20090 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:53:23.156203 20090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "50b8a91a2dff4a86b84ecd53c1df89ed"
format_stamp: "Formatted at 2026-06-19 16:53:23 on dist-test-slave-jsw9"
server_key: "15b6badb5306e8ed9712de48b820246e"
server_key_iv: "ec957ffd8ffbe5394e7037fca80c2b7d"
server_key_version: "encryptionkey@0"
I20260619 16:53:23.156291 20090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:23.209800 20090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:23.210270 20090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:23.210464 20090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:23.215345 20090 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39947
I20260619 16:53:23.215406 20156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39947 every 8 connection(s)
I20260619 16:53:23.215812 20090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:53:23.217029 20157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:23.218019 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 20090
I20260619 16:53:23.218139 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:53:23.218622 392 external_mini_cluster.cc:1428] Setting key 3f9c90f1792cc2c7bd38f462920a0e44
I20260619 16:53:23.220466 20157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50b8a91a2dff4a86b84ecd53c1df89ed: Bootstrap starting.
I20260619 16:53:23.221313 20157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50b8a91a2dff4a86b84ecd53c1df89ed: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:23.221676 20157 log.cc:824] T 00000000000000000000000000000000 P 50b8a91a2dff4a86b84ecd53c1df89ed: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:23.222786 20157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50b8a91a2dff4a86b84ecd53c1df89ed: No bootstrap required, opened a new log
I20260619 16:53:23.225215 20157 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50b8a91a2dff4a86b84ecd53c1df89ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50b8a91a2dff4a86b84ecd53c1df89ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39947 } }
I20260619 16:53:23.225437 20157 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50b8a91a2dff4a86b84ecd53c1df89ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:23.225478 20157 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50b8a91a2dff4a86b84ecd53c1df89ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50b8a91a2dff4a86b84ecd53c1df89ed, State: Initialized, Role: FOLLOWER
I20260619 16:53:23.225636 20157 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50b8a91a2dff4a86b84ecd53c1df89ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50b8a91a2dff4a86b84ecd53c1df89ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39947 } }
I20260619 16:53:23.225726 20157 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50b8a91a2dff4a86b84ecd53c1df89ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:53:23.225764 20157 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50b8a91a2dff4a86b84ecd53c1df89ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:53:23.225857 20157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50b8a91a2dff4a86b84ecd53c1df89ed [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:23.226745 20160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:23.219795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36936 (local address 127.0.98.62:39947)
0619 16:53:23.220060 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:23.220071 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:23.220109 (+ 38us) server_negotiation.cc:407] Connection header received
0619 16:53:23.220695 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:23.220707 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:23.220817 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:23.221025 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:23.221572 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:23.222866 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:23.223888 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:23.224580 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:23.224760 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:23.225852 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:23.225857 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:23.225857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:23.226115 (+ 258us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:23.226165 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:23.226209 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:23.226279 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:23.226377 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 16:53:23.226488 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":90,"threads_started":1}
I20260619 16:53:23.226876 20157 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50b8a91a2dff4a86b84ecd53c1df89ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50b8a91a2dff4a86b84ecd53c1df89ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39947 } }
I20260619 16:53:23.226994 20157 leader_election.cc:302] T 00000000000000000000000000000000 P 50b8a91a2dff4a86b84ecd53c1df89ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50b8a91a2dff4a86b84ecd53c1df89ed; no voters:
I20260619 16:53:23.227195 20157 leader_election.cc:288] T 00000000000000000000000000000000 P 50b8a91a2dff4a86b84ecd53c1df89ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:53:23.227239 20162 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50b8a91a2dff4a86b84ecd53c1df89ed [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:53:23.227357 20162 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50b8a91a2dff4a86b84ecd53c1df89ed [term 1 LEADER]: Becoming Leader. State: Replica: 50b8a91a2dff4a86b84ecd53c1df89ed, State: Running, Role: LEADER
I20260619 16:53:23.227499 20162 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50b8a91a2dff4a86b84ecd53c1df89ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50b8a91a2dff4a86b84ecd53c1df89ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39947 } }
I20260619 16:53:23.227533 20157 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50b8a91a2dff4a86b84ecd53c1df89ed [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:53:23.228010 20163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50b8a91a2dff4a86b84ecd53c1df89ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50b8a91a2dff4a86b84ecd53c1df89ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50b8a91a2dff4a86b84ecd53c1df89ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39947 } } }
I20260619 16:53:23.228021 20164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50b8a91a2dff4a86b84ecd53c1df89ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50b8a91a2dff4a86b84ecd53c1df89ed. Latest consensus state: current_term: 1 leader_uuid: "50b8a91a2dff4a86b84ecd53c1df89ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50b8a91a2dff4a86b84ecd53c1df89ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39947 } } }
I20260619 16:53:23.228229 20164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50b8a91a2dff4a86b84ecd53c1df89ed [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:23.228447 20163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50b8a91a2dff4a86b84ecd53c1df89ed [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:23.228480 20170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:53:23.229207 20170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:53:23.230916 20170 catalog_manager.cc:1347] Generated new cluster ID: 59395afd6b1a451887cfe6f7d424a5f4
I20260619 16:53:23.230975 20170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:53:23.241406 20170 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:53:23.242071 20170 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:53:23.245081 20170 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50b8a91a2dff4a86b84ecd53c1df89ed: Generated new TSK 0
I20260619 16:53:23.245288 20170 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:53:23.248423 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39947
--builtin_ntp_servers=127.0.98.20:43327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:53:23.353402 20181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:23.353679 20181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:23.353717 20181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:23.355089 20181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:23.355139 20181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:23.355176 20181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:53:23.356657 20181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:23.356950 20181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:23.357295 20181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:23.363951 20187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:23.363947 20189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:23.364059 20181 server_base.cc:1034] running on GCE node
W20260619 16:53:23.363963 20186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:23.364492 20181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:23.364766 20181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:23.365962 20181 hybrid_clock.cc:648] HybridClock initialized: now 1781888003365933 us; error 43 us; skew 500 ppm
I20260619 16:53:23.367676 20181 webserver.cc:460] Webserver started at http://127.0.98.1:35079/ using document root <none> and password file <none>
I20260619 16:53:23.367967 20181 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:23.368017 20181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:23.368160 20181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:23.369442 20181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-0/data/instance:
uuid: "f7b43748528a4c3fa57c52858b66fc16"
format_stamp: "Formatted at 2026-06-19 16:53:23 on dist-test-slave-jsw9"
server_key: "4a79e4f0d787b8e68155211e56f4f3c0"
server_key_iv: "cb84824318fb9e1e1ed135c9eb311b0c"
server_key_version: "encryptionkey@0"
I20260619 16:53:23.369822 20181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f7b43748528a4c3fa57c52858b66fc16"
format_stamp: "Formatted at 2026-06-19 16:53:23 on dist-test-slave-jsw9"
server_key: "4a79e4f0d787b8e68155211e56f4f3c0"
server_key_iv: "cb84824318fb9e1e1ed135c9eb311b0c"
server_key_version: "encryptionkey@0"
I20260619 16:53:23.371312 20181 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260619 16:53:23.372087 20195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:23.372239 20181 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 16:53:23.372332 20181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-0/wal
uuid: "f7b43748528a4c3fa57c52858b66fc16"
format_stamp: "Formatted at 2026-06-19 16:53:23 on dist-test-slave-jsw9"
server_key: "4a79e4f0d787b8e68155211e56f4f3c0"
server_key_iv: "cb84824318fb9e1e1ed135c9eb311b0c"
server_key_version: "encryptionkey@0"
I20260619 16:53:23.372411 20181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:23.393283 20181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:23.393716 20181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:23.393862 20181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:23.394126 20181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:53:23.394553 20181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:53:23.394589 20181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:23.394621 20181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:53:23.394644 20181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:23.401491 20181 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33591
I20260619 16:53:23.401549 20308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33591 every 8 connection(s)
I20260619 16:53:23.401932 20181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:53:23.407750 20160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:23.402818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56395 (local address 127.0.98.62:39947)
0619 16:53:23.402893 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:23.402895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:23.403193 (+ 298us) server_negotiation.cc:407] Connection header received
0619 16:53:23.403727 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:23.403728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:23.403746 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:23.403815 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:23.404594 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:23.405158 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:23.406290 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:23.406498 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:23.406554 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:23.407441 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:23.407442 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:23.407443 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:23.407448 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:23.407465 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:23.407469 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:23.407512 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:23.407614 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 16:53:23.407647 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 16:53:23.407866 20311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:23.402788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39947 (local address 127.0.98.1:56395)
0619 16:53:23.402994 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:23.403043 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:23.403620 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:23.403878 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:23.403882 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:23.403893 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:23.404470 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:23.404474 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:23.405263 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:23.405264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:23.406158 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:23.406160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:23.406574 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:23.406576 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:23.406708 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:23.407266 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:53:23.407266 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:23.407299 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:53:23.407356 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:23.407518 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:23.407521 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:23.407524 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:53:23.407574 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 16:53:23.407667 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":76,"threads_started":1}
I20260619 16:53:23.408389 20309 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39947
I20260619 16:53:23.408515 20309 heartbeater.cc:464] Registering TS with master...
I20260619 16:53:23.408648 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 20181
I20260619 16:53:23.408749 20309 heartbeater.cc:511] Master 127.0.98.62:39947 requested a full tablet report, sending...
I20260619 16:53:23.408776 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:53:23.409008 392 external_mini_cluster.cc:1428] Setting key 6053cedafdad92ccab7f0b347cded9ea
I20260619 16:53:23.409366 20121 ts_manager.cc:194] Registered new tserver with Master: f7b43748528a4c3fa57c52858b66fc16 (127.0.98.1:33591)
I20260619 16:53:23.410233 20121 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:56395
I20260619 16:53:23.410300 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39947
--builtin_ntp_servers=127.0.98.20:43327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:53:23.504235 20312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:23.504540 20312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:23.504580 20312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:23.506003 20312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:23.506058 20312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:23.506098 20312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:53:23.507769 20312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:23.508190 20312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:23.508610 20312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:23.514343 20317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:23.514345 20320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:23.514575 20312 server_base.cc:1034] running on GCE node
W20260619 16:53:23.514369 20318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:23.514889 20312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:23.515179 20312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:23.516359 20312 hybrid_clock.cc:648] HybridClock initialized: now 1781888003516341 us; error 40 us; skew 500 ppm
I20260619 16:53:23.517972 20312 webserver.cc:460] Webserver started at http://127.0.98.2:45917/ using document root <none> and password file <none>
I20260619 16:53:23.518285 20312 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:23.518347 20312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:23.518492 20312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:23.519733 20312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-1/data/instance:
uuid: "c7c70259c58c4d8aabd04332b0db337b"
format_stamp: "Formatted at 2026-06-19 16:53:23 on dist-test-slave-jsw9"
server_key: "d345e186eb609f47ddb679772d17f6e9"
server_key_iv: "9dbbd9dc8a36329251f62324546d3749"
server_key_version: "encryptionkey@0"
I20260619 16:53:23.520112 20312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c7c70259c58c4d8aabd04332b0db337b"
format_stamp: "Formatted at 2026-06-19 16:53:23 on dist-test-slave-jsw9"
server_key: "d345e186eb609f47ddb679772d17f6e9"
server_key_iv: "9dbbd9dc8a36329251f62324546d3749"
server_key_version: "encryptionkey@0"
I20260619 16:53:23.521641 20312 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 16:53:23.522593 20326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:23.522747 20312 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:53:23.522842 20312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-1/wal
uuid: "c7c70259c58c4d8aabd04332b0db337b"
format_stamp: "Formatted at 2026-06-19 16:53:23 on dist-test-slave-jsw9"
server_key: "d345e186eb609f47ddb679772d17f6e9"
server_key_iv: "9dbbd9dc8a36329251f62324546d3749"
server_key_version: "encryptionkey@0"
I20260619 16:53:23.522917 20312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:23.538066 20312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:23.538524 20312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:23.538681 20312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:23.538939 20312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:53:23.539338 20312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:53:23.539374 20312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:23.539415 20312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:53:23.539440 20312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:23.547410 20312 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40071
I20260619 16:53:23.547691 20439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40071 every 8 connection(s)
I20260619 16:53:23.547901 20312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:53:23.554028 20160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:23.548878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36871 (local address 127.0.98.62:39947)
0619 16:53:23.548947 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:23.548949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:23.549180 (+ 231us) server_negotiation.cc:407] Connection header received
0619 16:53:23.549805 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:23.549808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:23.549822 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:23.549881 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:23.550707 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:23.551257 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:23.552378 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:23.552592 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:23.552652 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:23.553687 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:23.553690 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:23.553690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:23.553699 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:23.553717 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:23.553722 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:23.553769 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:23.553876 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 16:53:23.553921 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 16:53:23.554330 20442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:23.548847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39947 (local address 127.0.98.2:36871)
0619 16:53:23.549044 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:23.549086 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:23.549688 (+ 602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:23.549969 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:23.549973 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:23.549983 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:23.550609 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:23.550615 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:23.551354 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:23.551355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:23.552283 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:23.552287 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:23.552675 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:23.552676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:23.552823 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:23.553507 (+ 684us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:53:23.553508 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:23.553544 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:53:23.553578 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:23.553798 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:23.553804 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:23.553807 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:53:23.553874 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 16:53:23.554009 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":78,"threads_started":1}
I20260619 16:53:23.555363 20440 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39947
I20260619 16:53:23.555491 20440 heartbeater.cc:464] Registering TS with master...
I20260619 16:53:23.555707 20440 heartbeater.cc:511] Master 127.0.98.62:39947 requested a full tablet report, sending...
I20260619 16:53:23.556368 20121 ts_manager.cc:194] Registered new tserver with Master: c7c70259c58c4d8aabd04332b0db337b (127.0.98.2:40071)
I20260619 16:53:23.556648 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 20312
I20260619 16:53:23.556816 20121 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:36871
I20260619 16:53:23.556895 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:53:23.557148 392 external_mini_cluster.cc:1428] Setting key f96fcbacc14ab56df79c535d073ddcc3
I20260619 16:53:23.558403 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39947
--builtin_ntp_servers=127.0.98.20:43327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:53:23.653030 20443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:23.653317 20443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:23.653345 20443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:23.654806 20443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:23.654857 20443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:23.654889 20443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:53:23.656664 20443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:23.656975 20443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:23.657424 20443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:23.663281 20449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:23.663283 20448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:23.663373 20451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:23.663602 20443 server_base.cc:1034] running on GCE node
I20260619 16:53:23.663841 20443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:23.664089 20443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:23.665288 20443 hybrid_clock.cc:648] HybridClock initialized: now 1781888003665276 us; error 32 us; skew 500 ppm
I20260619 16:53:23.666857 20443 webserver.cc:460] Webserver started at http://127.0.98.3:34457/ using document root <none> and password file <none>
I20260619 16:53:23.667114 20443 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:23.667167 20443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:23.667332 20443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:23.668599 20443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-2/data/instance:
uuid: "e9766dd1cc4946c79a021a95bfebd917"
format_stamp: "Formatted at 2026-06-19 16:53:23 on dist-test-slave-jsw9"
server_key: "dd688338b05aa14e67fd0e58859d3693"
server_key_iv: "27e940296344b165c138777f4c69cc06"
server_key_version: "encryptionkey@0"
I20260619 16:53:23.668967 20443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e9766dd1cc4946c79a021a95bfebd917"
format_stamp: "Formatted at 2026-06-19 16:53:23 on dist-test-slave-jsw9"
server_key: "dd688338b05aa14e67fd0e58859d3693"
server_key_iv: "27e940296344b165c138777f4c69cc06"
server_key_version: "encryptionkey@0"
I20260619 16:53:23.670449 20443 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260619 16:53:23.671247 20457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:23.671446 20443 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:53:23.671537 20443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-2/wal
uuid: "e9766dd1cc4946c79a021a95bfebd917"
format_stamp: "Formatted at 2026-06-19 16:53:23 on dist-test-slave-jsw9"
server_key: "dd688338b05aa14e67fd0e58859d3693"
server_key_iv: "27e940296344b165c138777f4c69cc06"
server_key_version: "encryptionkey@0"
I20260619 16:53:23.671603 20443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:23.689985 20443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:23.690428 20443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:23.690578 20443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:23.690867 20443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:53:23.691258 20443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:53:23.691293 20443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:23.691326 20443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:53:23.691352 20443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:23.697873 20443 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46253
I20260619 16:53:23.697927 20570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46253 every 8 connection(s)
I20260619 16:53:23.698364 20443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:53:23.704432 20160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:23.699333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55851 (local address 127.0.98.62:39947)
0619 16:53:23.699410 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:23.699413 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:23.699712 (+ 299us) server_negotiation.cc:407] Connection header received
0619 16:53:23.700320 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:23.700322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:23.700336 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:23.700395 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:23.701142 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:23.701702 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:23.702843 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:23.703056 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:23.703121 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:23.704040 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:23.704042 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:23.704042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:23.704049 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:23.704067 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:23.704072 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:23.704123 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:23.704305 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 16:53:23.704338 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 16:53:23.704620 20573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:23.699292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39947 (local address 127.0.98.3:55851)
0619 16:53:23.699575 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:23.699619 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:23.700198 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:23.700458 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:23.700462 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:23.700472 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:23.701031 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:23.701036 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:23.701804 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:23.701805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:23.702745 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:23.702748 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:23.703127 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:23.703128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:23.703255 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:23.703871 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:53:23.703872 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:23.703907 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:53:23.703935 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:23.704153 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:23.704159 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:23.704162 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:53:23.704241 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 16:53:23.704356 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":103,"threads_started":1}
I20260619 16:53:23.705119 20571 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39947
I20260619 16:53:23.705220 20571 heartbeater.cc:464] Registering TS with master...
I20260619 16:53:23.705399 20571 heartbeater.cc:511] Master 127.0.98.62:39947 requested a full tablet report, sending...
I20260619 16:53:23.705708 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 20443
I20260619 16:53:23.705808 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:53:23.705847 20121 ts_manager.cc:194] Registered new tserver with Master: e9766dd1cc4946c79a021a95bfebd917 (127.0.98.3:46253)
I20260619 16:53:23.706034 392 external_mini_cluster.cc:1428] Setting key f742a9129a708b644dd72472afb71cb9
I20260619 16:53:23.706358 20121 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:55851
I20260619 16:53:23.706998 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:53:23.711294 20160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:23.708468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36952 (local address 127.0.98.62:39947)
0619 16:53:23.708516 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:23.708519 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:23.708633 (+ 114us) server_negotiation.cc:407] Connection header received
0619 16:53:23.708676 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:23.708676 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:23.708686 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:23.708728 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:23.708852 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:23.709361 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:23.710123 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:23.710406 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:23.710468 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:23.711095 (+ 627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:23.711097 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:23.711097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:23.711101 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:23.711115 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:23.711119 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:23.711156 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:23.711194 (+ 38us) server_negotiation.cc:299] Negotiation successful
0619 16:53:23.711217 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 16:53:23.712870 20121 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:53:23.714860 20121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:53:23.722182 20582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:23.717570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48228 (local address 127.0.98.3:46253)
0619 16:53:23.718577 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:23.718582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:23.718593 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:53:23.718637 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:23.718638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:23.718663 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:23.718741 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:23.718978 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:23.720058 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:23.721126 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:23.721698 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:23.721767 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:23.721889 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:23.721967 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 16:53:23.722012 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":946,"thread_start_us":111,"threads_started":1}
I20260619 16:53:23.722110 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:23.717429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46253 (local address 127.0.0.1:48228)
0619 16:53:23.718088 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:23.718100 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:23.718161 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:23.718821 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:23.718821 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:23.718824 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:23.718886 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:23.718887 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:23.720125 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:23.720126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:23.721040 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:23.721043 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:23.721780 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:23.721781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:23.721860 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:23.721875 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 16:53:23.721950 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 16:53:23.722012 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":107,"threads_started":1}
I20260619 16:53:23.722329 20581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:23.717592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42244 (local address 127.0.98.2:40071)
0619 16:53:23.717891 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:23.717897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:23.718041 (+ 144us) server_negotiation.cc:407] Connection header received
0619 16:53:23.718149 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:23.718152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:23.718170 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:23.718312 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:23.718555 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:23.719479 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:23.720969 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:23.721732 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:23.721805 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:23.721892 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:23.722153 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 16:53:23.722228 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":213,"threads_started":1}
I20260619 16:53:23.722596 20579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:23.717935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33591 (local address 127.0.0.1:36474)
0619 16:53:23.718168 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:23.718177 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:23.718261 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:23.718610 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:23.718610 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:23.718612 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:23.718663 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:23.718664 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:23.719979 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:23.719980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:23.721116 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:23.721119 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:23.722015 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:23.722015 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:23.722067 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:23.722070 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:53:23.722095 (+ 25us) client_negotiation.cc:240] Negotiation successful
0619 16:53:23.722132 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":23,"thread_start_us":92,"threads_started":1}
I20260619 16:53:23.722846 20580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:23.717422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40071 (local address 127.0.0.1:42244)
0619 16:53:23.717923 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:23.717970 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:23.718054 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:23.718326 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:23.718331 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:23.718334 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:23.718457 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:23.718460 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:23.719554 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:23.719555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:23.720873 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:23.720876 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:23.721932 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:23.721934 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:23.721995 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:23.721999 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:53:23.722042 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 16:53:23.722091 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":107,"threads_started":1}
I20260619 16:53:23.723307 20584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:23.718233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36474 (local address 127.0.98.1:33591)
0619 16:53:23.718443 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:23.718446 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:23.718455 (+ 9us) server_negotiation.cc:407] Connection header received
0619 16:53:23.718494 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:23.718495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:23.718511 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:23.718730 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:23.718805 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:23.719891 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:23.721185 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:23.721931 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:23.722174 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:23.722323 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:23.722972 (+ 649us) server_negotiation.cc:299] Negotiation successful
0619 16:53:23.723214 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":119,"threads_started":1}
I20260619 16:53:23.723342 20505 tablet_service.cc:1467] Processing CreateTablet for tablet f8391ef0b55849d0b7258f463f088e47 (DEFAULT_TABLE table=test-table [id=e8e42d476fa34cdc8f401346a222517e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:53:23.723603 20374 tablet_service.cc:1467] Processing CreateTablet for tablet f8391ef0b55849d0b7258f463f088e47 (DEFAULT_TABLE table=test-table [id=e8e42d476fa34cdc8f401346a222517e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:53:23.723788 20505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8391ef0b55849d0b7258f463f088e47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:23.723932 20374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8391ef0b55849d0b7258f463f088e47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:23.727106 20585 tablet_bootstrap.cc:492] T f8391ef0b55849d0b7258f463f088e47 P c7c70259c58c4d8aabd04332b0db337b: Bootstrap starting.
I20260619 16:53:23.727566 20586 tablet_bootstrap.cc:492] T f8391ef0b55849d0b7258f463f088e47 P e9766dd1cc4946c79a021a95bfebd917: Bootstrap starting.
I20260619 16:53:23.728096 20585 tablet_bootstrap.cc:654] T f8391ef0b55849d0b7258f463f088e47 P c7c70259c58c4d8aabd04332b0db337b: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:23.728351 20586 tablet_bootstrap.cc:654] T f8391ef0b55849d0b7258f463f088e47 P e9766dd1cc4946c79a021a95bfebd917: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:23.728219 20243 tablet_service.cc:1467] Processing CreateTablet for tablet f8391ef0b55849d0b7258f463f088e47 (DEFAULT_TABLE table=test-table [id=e8e42d476fa34cdc8f401346a222517e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:53:23.728534 20585 log.cc:824] T f8391ef0b55849d0b7258f463f088e47 P c7c70259c58c4d8aabd04332b0db337b: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:23.728681 20586 log.cc:824] T f8391ef0b55849d0b7258f463f088e47 P e9766dd1cc4946c79a021a95bfebd917: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:23.728696 20243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8391ef0b55849d0b7258f463f088e47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:23.729410 20585 tablet_bootstrap.cc:492] T f8391ef0b55849d0b7258f463f088e47 P c7c70259c58c4d8aabd04332b0db337b: No bootstrap required, opened a new log
I20260619 16:53:23.729485 20585 ts_tablet_manager.cc:1397] T f8391ef0b55849d0b7258f463f088e47 P c7c70259c58c4d8aabd04332b0db337b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:53:23.732312 20585 raft_consensus.cc:348] T f8391ef0b55849d0b7258f463f088e47 P c7c70259c58c4d8aabd04332b0db337b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9766dd1cc4946c79a021a95bfebd917" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46253 } } peers { permanent_uuid: "f7b43748528a4c3fa57c52858b66fc16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } } peers { permanent_uuid: "c7c70259c58c4d8aabd04332b0db337b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40071 } }
I20260619 16:53:23.732522 20585 raft_consensus.cc:374] T f8391ef0b55849d0b7258f463f088e47 P c7c70259c58c4d8aabd04332b0db337b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:23.732559 20585 raft_consensus.cc:729] T f8391ef0b55849d0b7258f463f088e47 P c7c70259c58c4d8aabd04332b0db337b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7c70259c58c4d8aabd04332b0db337b, State: Initialized, Role: FOLLOWER
I20260619 16:53:23.732651 20589 tablet_bootstrap.cc:492] T f8391ef0b55849d0b7258f463f088e47 P f7b43748528a4c3fa57c52858b66fc16: Bootstrap starting.
I20260619 16:53:23.732729 20585 consensus_queue.cc:260] T f8391ef0b55849d0b7258f463f088e47 P c7c70259c58c4d8aabd04332b0db337b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9766dd1cc4946c79a021a95bfebd917" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46253 } } peers { permanent_uuid: "f7b43748528a4c3fa57c52858b66fc16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } } peers { permanent_uuid: "c7c70259c58c4d8aabd04332b0db337b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40071 } }
I20260619 16:53:23.733022 20585 ts_tablet_manager.cc:1428] T f8391ef0b55849d0b7258f463f088e47 P c7c70259c58c4d8aabd04332b0db337b: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260619 16:53:23.733171 20440 heartbeater.cc:503] Master 127.0.98.62:39947 was elected leader, sending a full tablet report...
I20260619 16:53:23.733362 20589 tablet_bootstrap.cc:654] T f8391ef0b55849d0b7258f463f088e47 P f7b43748528a4c3fa57c52858b66fc16: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:23.733553 20586 tablet_bootstrap.cc:492] T f8391ef0b55849d0b7258f463f088e47 P e9766dd1cc4946c79a021a95bfebd917: No bootstrap required, opened a new log
I20260619 16:53:23.733637 20586 ts_tablet_manager.cc:1397] T f8391ef0b55849d0b7258f463f088e47 P e9766dd1cc4946c79a021a95bfebd917: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260619 16:53:23.733819 20589 log.cc:824] T f8391ef0b55849d0b7258f463f088e47 P f7b43748528a4c3fa57c52858b66fc16: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:23.734623 20589 tablet_bootstrap.cc:492] T f8391ef0b55849d0b7258f463f088e47 P f7b43748528a4c3fa57c52858b66fc16: No bootstrap required, opened a new log
I20260619 16:53:23.734692 20589 ts_tablet_manager.cc:1397] T f8391ef0b55849d0b7258f463f088e47 P f7b43748528a4c3fa57c52858b66fc16: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:53:23.735986 20586 raft_consensus.cc:348] T f8391ef0b55849d0b7258f463f088e47 P e9766dd1cc4946c79a021a95bfebd917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9766dd1cc4946c79a021a95bfebd917" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46253 } } peers { permanent_uuid: "f7b43748528a4c3fa57c52858b66fc16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } } peers { permanent_uuid: "c7c70259c58c4d8aabd04332b0db337b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40071 } }
I20260619 16:53:23.736145 20586 raft_consensus.cc:374] T f8391ef0b55849d0b7258f463f088e47 P e9766dd1cc4946c79a021a95bfebd917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:23.736176 20586 raft_consensus.cc:729] T f8391ef0b55849d0b7258f463f088e47 P e9766dd1cc4946c79a021a95bfebd917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9766dd1cc4946c79a021a95bfebd917, State: Initialized, Role: FOLLOWER
I20260619 16:53:23.736308 20586 consensus_queue.cc:260] T f8391ef0b55849d0b7258f463f088e47 P e9766dd1cc4946c79a021a95bfebd917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9766dd1cc4946c79a021a95bfebd917" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46253 } } peers { permanent_uuid: "f7b43748528a4c3fa57c52858b66fc16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } } peers { permanent_uuid: "c7c70259c58c4d8aabd04332b0db337b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40071 } }
I20260619 16:53:23.736582 20586 ts_tablet_manager.cc:1428] T f8391ef0b55849d0b7258f463f088e47 P e9766dd1cc4946c79a021a95bfebd917: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 16:53:23.736649 20571 heartbeater.cc:503] Master 127.0.98.62:39947 was elected leader, sending a full tablet report...
I20260619 16:53:23.736737 20589 raft_consensus.cc:348] T f8391ef0b55849d0b7258f463f088e47 P f7b43748528a4c3fa57c52858b66fc16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9766dd1cc4946c79a021a95bfebd917" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46253 } } peers { permanent_uuid: "f7b43748528a4c3fa57c52858b66fc16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } } peers { permanent_uuid: "c7c70259c58c4d8aabd04332b0db337b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40071 } }
I20260619 16:53:23.736856 20589 raft_consensus.cc:374] T f8391ef0b55849d0b7258f463f088e47 P f7b43748528a4c3fa57c52858b66fc16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:23.736896 20589 raft_consensus.cc:729] T f8391ef0b55849d0b7258f463f088e47 P f7b43748528a4c3fa57c52858b66fc16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7b43748528a4c3fa57c52858b66fc16, State: Initialized, Role: FOLLOWER
I20260619 16:53:23.737010 20589 consensus_queue.cc:260] T f8391ef0b55849d0b7258f463f088e47 P f7b43748528a4c3fa57c52858b66fc16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9766dd1cc4946c79a021a95bfebd917" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46253 } } peers { permanent_uuid: "f7b43748528a4c3fa57c52858b66fc16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } } peers { permanent_uuid: "c7c70259c58c4d8aabd04332b0db337b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40071 } }
I20260619 16:53:23.737273 20589 ts_tablet_manager.cc:1428] T f8391ef0b55849d0b7258f463f088e47 P f7b43748528a4c3fa57c52858b66fc16: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 16:53:23.737339 20309 heartbeater.cc:503] Master 127.0.98.62:39947 was elected leader, sending a full tablet report...
W20260619 16:53:23.798606 20441 tablet.cc:2369] T f8391ef0b55849d0b7258f463f088e47 P c7c70259c58c4d8aabd04332b0db337b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:53:23.802026 20592 raft_consensus.cc:482] T f8391ef0b55849d0b7258f463f088e47 P e9766dd1cc4946c79a021a95bfebd917 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:53:23.802229 20592 raft_consensus.cc:504] T f8391ef0b55849d0b7258f463f088e47 P e9766dd1cc4946c79a021a95bfebd917 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9766dd1cc4946c79a021a95bfebd917" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46253 } } peers { permanent_uuid: "f7b43748528a4c3fa57c52858b66fc16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } } peers { permanent_uuid: "c7c70259c58c4d8aabd04332b0db337b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40071 } }
I20260619 16:53:23.802721 20592 leader_election.cc:288] T f8391ef0b55849d0b7258f463f088e47 P e9766dd1cc4946c79a021a95bfebd917 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7b43748528a4c3fa57c52858b66fc16 (127.0.98.1:33591), c7c70259c58c4d8aabd04332b0db337b (127.0.98.2:40071)
I20260619 16:53:23.805779 20581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:23.803104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60907 (local address 127.0.98.2:40071)
0619 16:53:23.803155 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:23.803157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:23.803320 (+ 163us) server_negotiation.cc:407] Connection header received
0619 16:53:23.803358 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:23.803359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:23.803374 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:23.803420 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:23.803566 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:23.804045 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:23.805053 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:23.805436 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:23.805497 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:23.805510 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:23.805648 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 16:53:23.805693 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 16:53:23.805775 20584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:23.803104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37113 (local address 127.0.98.1:33591)
0619 16:53:23.803155 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:23.803157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:23.803167 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:53:23.803202 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:23.803203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:23.803216 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:23.803319 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:53:23.803479 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:23.803998 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:23.805051 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:23.805439 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:23.805497 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:23.805510 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:23.805649 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 16:53:23.805693 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 16:53:23.805781 20573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:23.802990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33591 (local address 127.0.98.3:37113)
0619 16:53:23.803037 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:23.803052 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:23.803102 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:23.803305 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:23.803307 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:23.803310 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:23.803405 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:23.803408 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:23.804075 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:23.804076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:23.804955 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:23.804958 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:23.805511 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:23.805512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:23.805584 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:23.805587 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:53:23.805633 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 16:53:23.805690 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":40}
I20260619 16:53:23.805804 20594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:23.802990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40071 (local address 127.0.98.3:60907)
0619 16:53:23.803213 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:23.803221 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:23.803305 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:23.803431 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:23.803432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:23.803434 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:53:23.803505 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:23.803507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:23.804127 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:23.804128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:23.804966 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:23.804968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:23.805514 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:23.805515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:23.805582 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:23.805587 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:53:23.805632 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 16:53:23.805685 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":75,"thread_start_us":124,"threads_started":1}
I20260619 16:53:23.806267 20263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8391ef0b55849d0b7258f463f088e47" candidate_uuid: "e9766dd1cc4946c79a021a95bfebd917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7b43748528a4c3fa57c52858b66fc16" is_pre_election: true
I20260619 16:53:23.806264 20394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8391ef0b55849d0b7258f463f088e47" candidate_uuid: "e9766dd1cc4946c79a021a95bfebd917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7c70259c58c4d8aabd04332b0db337b" is_pre_election: true
I20260619 16:53:23.806448 20394 raft_consensus.cc:2454] T f8391ef0b55849d0b7258f463f088e47 P c7c70259c58c4d8aabd04332b0db337b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9766dd1cc4946c79a021a95bfebd917 in term 0.
I20260619 16:53:23.806454 20263 raft_consensus.cc:2454] T f8391ef0b55849d0b7258f463f088e47 P f7b43748528a4c3fa57c52858b66fc16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9766dd1cc4946c79a021a95bfebd917 in term 0.
I20260619 16:53:23.806689 20458 leader_election.cc:302] T f8391ef0b55849d0b7258f463f088e47 P e9766dd1cc4946c79a021a95bfebd917 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7c70259c58c4d8aabd04332b0db337b, e9766dd1cc4946c79a021a95bfebd917; no voters:
I20260619 16:53:23.806820 20592 raft_consensus.cc:2781] T f8391ef0b55849d0b7258f463f088e47 P e9766dd1cc4946c79a021a95bfebd917 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:53:23.806886 20592 raft_consensus.cc:482] T f8391ef0b55849d0b7258f463f088e47 P e9766dd1cc4946c79a021a95bfebd917 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:53:23.806910 20592 raft_consensus.cc:3037] T f8391ef0b55849d0b7258f463f088e47 P e9766dd1cc4946c79a021a95bfebd917 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:23.807789 20592 raft_consensus.cc:504] T f8391ef0b55849d0b7258f463f088e47 P e9766dd1cc4946c79a021a95bfebd917 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9766dd1cc4946c79a021a95bfebd917" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46253 } } peers { permanent_uuid: "f7b43748528a4c3fa57c52858b66fc16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } } peers { permanent_uuid: "c7c70259c58c4d8aabd04332b0db337b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40071 } }
I20260619 16:53:23.807935 20592 leader_election.cc:288] T f8391ef0b55849d0b7258f463f088e47 P e9766dd1cc4946c79a021a95bfebd917 [CANDIDATE]: Term 1 election: Requested vote from peers f7b43748528a4c3fa57c52858b66fc16 (127.0.98.1:33591), c7c70259c58c4d8aabd04332b0db337b (127.0.98.2:40071)
I20260619 16:53:23.808125 20394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8391ef0b55849d0b7258f463f088e47" candidate_uuid: "e9766dd1cc4946c79a021a95bfebd917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7c70259c58c4d8aabd04332b0db337b"
I20260619 16:53:23.808151 20263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8391ef0b55849d0b7258f463f088e47" candidate_uuid: "e9766dd1cc4946c79a021a95bfebd917" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7b43748528a4c3fa57c52858b66fc16"
I20260619 16:53:23.808209 20394 raft_consensus.cc:3037] T f8391ef0b55849d0b7258f463f088e47 P c7c70259c58c4d8aabd04332b0db337b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:23.808262 20263 raft_consensus.cc:3037] T f8391ef0b55849d0b7258f463f088e47 P f7b43748528a4c3fa57c52858b66fc16 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:23.809103 20394 raft_consensus.cc:2454] T f8391ef0b55849d0b7258f463f088e47 P c7c70259c58c4d8aabd04332b0db337b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9766dd1cc4946c79a021a95bfebd917 in term 1.
I20260619 16:53:23.809185 20263 raft_consensus.cc:2454] T f8391ef0b55849d0b7258f463f088e47 P f7b43748528a4c3fa57c52858b66fc16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9766dd1cc4946c79a021a95bfebd917 in term 1.
I20260619 16:53:23.809366 20460 leader_election.cc:302] T f8391ef0b55849d0b7258f463f088e47 P e9766dd1cc4946c79a021a95bfebd917 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e9766dd1cc4946c79a021a95bfebd917, f7b43748528a4c3fa57c52858b66fc16; no voters:
I20260619 16:53:23.809486 20592 raft_consensus.cc:2781] T f8391ef0b55849d0b7258f463f088e47 P e9766dd1cc4946c79a021a95bfebd917 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:53:23.809684 20592 raft_consensus.cc:686] T f8391ef0b55849d0b7258f463f088e47 P e9766dd1cc4946c79a021a95bfebd917 [term 1 LEADER]: Becoming Leader. State: Replica: e9766dd1cc4946c79a021a95bfebd917, State: Running, Role: LEADER
I20260619 16:53:23.809852 20592 consensus_queue.cc:237] T f8391ef0b55849d0b7258f463f088e47 P e9766dd1cc4946c79a021a95bfebd917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9766dd1cc4946c79a021a95bfebd917" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46253 } } peers { permanent_uuid: "f7b43748528a4c3fa57c52858b66fc16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } } peers { permanent_uuid: "c7c70259c58c4d8aabd04332b0db337b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40071 } }
I20260619 16:53:23.810905 20121 catalog_manager.cc:5495] T f8391ef0b55849d0b7258f463f088e47 P e9766dd1cc4946c79a021a95bfebd917 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9766dd1cc4946c79a021a95bfebd917 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e9766dd1cc4946c79a021a95bfebd917" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9766dd1cc4946c79a021a95bfebd917" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7b43748528a4c3fa57c52858b66fc16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7c70259c58c4d8aabd04332b0db337b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40071 } health_report { overall_health: UNKNOWN } } }
I20260619 16:53:23.838989 20582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:23.836179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48234 (local address 127.0.98.3:46253)
0619 16:53:23.836248 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:23.836249 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:23.836259 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:53:23.836303 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:23.836304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:23.836325 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:23.836389 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:53:23.836520 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:23.837037 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:23.837984 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:23.838259 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:23.838337 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:23.838553 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:53:23.838695 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:53:23.838775 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:23.838841 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 16:53:23.838862 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 16:53:23.841339 20263 raft_consensus.cc:1261] T f8391ef0b55849d0b7258f463f088e47 P f7b43748528a4c3fa57c52858b66fc16 [term 1 FOLLOWER]: Refusing update from remote peer e9766dd1cc4946c79a021a95bfebd917: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:53:23.841348 20394 raft_consensus.cc:1261] T f8391ef0b55849d0b7258f463f088e47 P c7c70259c58c4d8aabd04332b0db337b [term 1 FOLLOWER]: Refusing update from remote peer e9766dd1cc4946c79a021a95bfebd917: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:53:23.841755 20595 consensus_queue.cc:1035] T f8391ef0b55849d0b7258f463f088e47 P e9766dd1cc4946c79a021a95bfebd917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7b43748528a4c3fa57c52858b66fc16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:53:23.841939 20592 consensus_queue.cc:1035] T f8391ef0b55849d0b7258f463f088e47 P e9766dd1cc4946c79a021a95bfebd917 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7c70259c58c4d8aabd04332b0db337b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:53:23.845681 20599 mvcc.cc:205] Tried to move back new op lower bound from 7298613263730778112 to 7298613263606099968. Current Snapshot: MvccSnapshot[applied={T|T < 7298613263730778112}]
I20260619 16:53:23.848095 20121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:53:23.848240 20121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:53:23.849211 20121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 50b8a91a2dff4a86b84ecd53c1df89ed: Sending DeleteTablet for 3 replicas of tablet f8391ef0b55849d0b7258f463f088e47
I20260619 16:53:23.849659 20374 tablet_service.cc:1514] Processing DeleteTablet for tablet f8391ef0b55849d0b7258f463f088e47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:53:23 UTC) from {username='slave'} at 127.0.0.1:42244
I20260619 16:53:23.849694 20243 tablet_service.cc:1514] Processing DeleteTablet for tablet f8391ef0b55849d0b7258f463f088e47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:53:23 UTC) from {username='slave'} at 127.0.0.1:36474
I20260619 16:53:23.849982 20607 tablet_replica.cc:330] T f8391ef0b55849d0b7258f463f088e47 P f7b43748528a4c3fa57c52858b66fc16: stopping tablet replica
I20260619 16:53:23.850000 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 20181
I20260619 16:53:23.850013 20505 tablet_service.cc:1514] Processing DeleteTablet for tablet f8391ef0b55849d0b7258f463f088e47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:53:23 UTC) from {username='slave'} at 127.0.0.1:48228
I20260619 16:53:23.850152 20608 tablet_replica.cc:330] T f8391ef0b55849d0b7258f463f088e47 P c7c70259c58c4d8aabd04332b0db337b: stopping tablet replica
I20260619 16:53:23.850303 20608 raft_consensus.cc:2229] T f8391ef0b55849d0b7258f463f088e47 P c7c70259c58c4d8aabd04332b0db337b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:53:23.850430 20607 raft_consensus.cc:2229] T f8391ef0b55849d0b7258f463f088e47 P f7b43748528a4c3fa57c52858b66fc16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:53:23.850636 20608 raft_consensus.cc:2258] T f8391ef0b55849d0b7258f463f088e47 P c7c70259c58c4d8aabd04332b0db337b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:53:23.851197 20608 ts_tablet_manager.cc:1905] T f8391ef0b55849d0b7258f463f088e47 P c7c70259c58c4d8aabd04332b0db337b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:53:23.852854 20608 ts_tablet_manager.cc:1918] T f8391ef0b55849d0b7258f463f088e47 P c7c70259c58c4d8aabd04332b0db337b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:53:23.852933 20608 log.cc:1196] T f8391ef0b55849d0b7258f463f088e47 P c7c70259c58c4d8aabd04332b0db337b: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-1/wal/wals/f8391ef0b55849d0b7258f463f088e47
I20260619 16:53:23.853160 20608 ts_tablet_manager.cc:1939] T f8391ef0b55849d0b7258f463f088e47 P c7c70259c58c4d8aabd04332b0db337b: Deleting consensus metadata
I20260619 16:53:23.853681 20105 catalog_manager.cc:4841] TS c7c70259c58c4d8aabd04332b0db337b (127.0.98.2:40071): tablet f8391ef0b55849d0b7258f463f088e47 (table test-table [id=e8e42d476fa34cdc8f401346a222517e]) successfully deleted
I20260619 16:53:23.855513 20609 tablet_replica.cc:330] T f8391ef0b55849d0b7258f463f088e47 P e9766dd1cc4946c79a021a95bfebd917: stopping tablet replica
I20260619 16:53:23.855679 20609 raft_consensus.cc:2229] T f8391ef0b55849d0b7258f463f088e47 P e9766dd1cc4946c79a021a95bfebd917 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:53:23.855836 20609 raft_consensus.cc:2258] T f8391ef0b55849d0b7258f463f088e47 P e9766dd1cc4946c79a021a95bfebd917 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:53:23.856365 20609 ts_tablet_manager.cc:1905] T f8391ef0b55849d0b7258f463f088e47 P e9766dd1cc4946c79a021a95bfebd917: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 16:53:23.857316 20107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33591 (error 108)
I20260619 16:53:23.857951 20580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:23.857785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33591 (local address 127.0.0.1:36482)
0619 16:53:23.857856 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:23.857908 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 16:53:23.858069 20107 catalog_manager.cc:4584] TS f7b43748528a4c3fa57c52858b66fc16 (127.0.98.1:33591): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f8391ef0b55849d0b7258f463f088e47: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33591: connect: Connection refused (error 111)
I20260619 16:53:23.858451 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 20312
I20260619 16:53:23.858610 20609 ts_tablet_manager.cc:1918] T f8391ef0b55849d0b7258f463f088e47 P e9766dd1cc4946c79a021a95bfebd917: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:53:23.858688 20609 log.cc:1196] T f8391ef0b55849d0b7258f463f088e47 P e9766dd1cc4946c79a021a95bfebd917: Deleting WAL directory at /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781887741903642-392-0/minicluster-data/ts-2/wal/wals/f8391ef0b55849d0b7258f463f088e47
I20260619 16:53:23.859537 20609 ts_tablet_manager.cc:1939] T f8391ef0b55849d0b7258f463f088e47 P e9766dd1cc4946c79a021a95bfebd917: Deleting consensus metadata
I20260619 16:53:23.861440 20107 catalog_manager.cc:4841] TS e9766dd1cc4946c79a021a95bfebd917 (127.0.98.3:46253): tablet f8391ef0b55849d0b7258f463f088e47 (table test-table [id=e8e42d476fa34cdc8f401346a222517e]) successfully deleted
I20260619 16:53:23.866066 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 20443
I20260619 16:53:23.871148 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 20090
2026-06-19T16:53:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (859 ms)
[----------] 6 tests from EncryptionPolicyTest (9327 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:53:23 dist-test-slave-jsw9 krb5kdc[20613](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:53:23 dist-test-slave-jsw9 krb5kdc[20613](info): set up 2 sockets
Jun 19 16:53:23 dist-test-slave-jsw9 krb5kdc[20613](info): commencing operation
krb5kdc: starting...
W20260619 16:53:24.846634 392 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:53:24 dist-test-slave-jsw9 krb5kdc[20613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:53:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:53:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:53:24.977892 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46041
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:53:25.076289 20626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:25.076577 20626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:25.076606 20626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:25.077976 20626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:53:25.078035 20626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:25.078050 20626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:25.078063 20626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:53:25.078073 20626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:53:25.079911 20626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46041
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:25.080189 20626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:25.080427 20626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:25.086655 20631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:25.086648 20634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:25.086920 20626 server_base.cc:1034] running on GCE node
W20260619 16:53:25.086639 20632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:25.087430 20626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:25.087766 20626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:25.088948 20626 hybrid_clock.cc:648] HybridClock initialized: now 1781888005088924 us; error 38 us; skew 500 ppm
Jun 19 16:53:25 dist-test-slave-jsw9 krb5kdc[20613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:25.097513 20626 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:53:25.098129 20626 webserver.cc:460] Webserver started at http://127.0.0.1:40903/ using document root <none> and password file <none>
I20260619 16:53:25.098425 20626 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:25.098481 20626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:25.098615 20626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:25.099859 20626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "09a76dcc66074a79b4f886fe359cb178"
format_stamp: "Formatted at 2026-06-19 16:53:25 on dist-test-slave-jsw9"
server_key: "5f1d974f29781387708aa791cc099b19"
server_key_iv: "0cd13c56e3e85a58ab80da5fbea3ecf7"
server_key_version: "encryptionkey@0"
I20260619 16:53:25.100229 20626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "09a76dcc66074a79b4f886fe359cb178"
format_stamp: "Formatted at 2026-06-19 16:53:25 on dist-test-slave-jsw9"
server_key: "5f1d974f29781387708aa791cc099b19"
server_key_iv: "0cd13c56e3e85a58ab80da5fbea3ecf7"
server_key_version: "encryptionkey@0"
I20260619 16:53:25.101694 20626 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 16:53:25.102579 20641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:25.102748 20626 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:53:25.102843 20626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "09a76dcc66074a79b4f886fe359cb178"
format_stamp: "Formatted at 2026-06-19 16:53:25 on dist-test-slave-jsw9"
server_key: "5f1d974f29781387708aa791cc099b19"
server_key_iv: "0cd13c56e3e85a58ab80da5fbea3ecf7"
server_key_version: "encryptionkey@0"
I20260619 16:53:25.102924 20626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:25.116925 20626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:25.119380 20626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:25.119524 20626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:25.124075 20626 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46041
I20260619 16:53:25.124119 20693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46041 every 8 connection(s)
I20260619 16:53:25.124536 20626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:53:25.125648 20694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:25.128602 20694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09a76dcc66074a79b4f886fe359cb178: Bootstrap starting.
I20260619 16:53:25.129309 20694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09a76dcc66074a79b4f886fe359cb178: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:25.129618 20694 log.cc:824] T 00000000000000000000000000000000 P 09a76dcc66074a79b4f886fe359cb178: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:25.130551 20694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09a76dcc66074a79b4f886fe359cb178: No bootstrap required, opened a new log
I20260619 16:53:25.132436 20694 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09a76dcc66074a79b4f886fe359cb178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09a76dcc66074a79b4f886fe359cb178" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46041 } }
I20260619 16:53:25.132611 20694 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09a76dcc66074a79b4f886fe359cb178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:25.132635 20694 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09a76dcc66074a79b4f886fe359cb178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a76dcc66074a79b4f886fe359cb178, State: Initialized, Role: FOLLOWER
I20260619 16:53:25.132797 20694 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09a76dcc66074a79b4f886fe359cb178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09a76dcc66074a79b4f886fe359cb178" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46041 } }
I20260619 16:53:25.132884 20694 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09a76dcc66074a79b4f886fe359cb178 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:53:25.132932 20694 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09a76dcc66074a79b4f886fe359cb178 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:53:25.132977 20694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09a76dcc66074a79b4f886fe359cb178 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:25.133651 20694 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09a76dcc66074a79b4f886fe359cb178 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09a76dcc66074a79b4f886fe359cb178" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46041 } }
I20260619 16:53:25.133754 20694 leader_election.cc:302] T 00000000000000000000000000000000 P 09a76dcc66074a79b4f886fe359cb178 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09a76dcc66074a79b4f886fe359cb178; no voters:
I20260619 16:53:25.133971 20694 leader_election.cc:288] T 00000000000000000000000000000000 P 09a76dcc66074a79b4f886fe359cb178 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:53:25.134020 20697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09a76dcc66074a79b4f886fe359cb178 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:53:25.134210 20697 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09a76dcc66074a79b4f886fe359cb178 [term 1 LEADER]: Becoming Leader. State: Replica: 09a76dcc66074a79b4f886fe359cb178, State: Running, Role: LEADER
I20260619 16:53:25.134299 20694 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09a76dcc66074a79b4f886fe359cb178 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:53:25.134317 20697 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09a76dcc66074a79b4f886fe359cb178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09a76dcc66074a79b4f886fe359cb178" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46041 } }
I20260619 16:53:25.134706 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 20626
I20260619 16:53:25.134835 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:53:25.134908 20699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09a76dcc66074a79b4f886fe359cb178 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09a76dcc66074a79b4f886fe359cb178. Latest consensus state: current_term: 1 leader_uuid: "09a76dcc66074a79b4f886fe359cb178" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09a76dcc66074a79b4f886fe359cb178" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46041 } } }
I20260619 16:53:25.134982 20699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09a76dcc66074a79b4f886fe359cb178 [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:25.135077 392 external_mini_cluster.cc:1428] Setting key 7537bd65035239ad5aa08dbbe623b133
I20260619 16:53:25.135259 20698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09a76dcc66074a79b4f886fe359cb178 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09a76dcc66074a79b4f886fe359cb178" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09a76dcc66074a79b4f886fe359cb178" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46041 } } }
I20260619 16:53:25.135318 20698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09a76dcc66074a79b4f886fe359cb178 [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:25.135378 20705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:53:25.136021 20705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:53:25.137943 20705 catalog_manager.cc:1347] Generated new cluster ID: 919d6195afe34d348aa0f8e1be58a398
I20260619 16:53:25.137997 20705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 16:53:25 dist-test-slave-jsw9 krb5kdc[20613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 16:53:25.148277 20715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:25.135879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39209 (local address 127.0.0.1:46041)
0619 16:53:25.136053 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:25.136065 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:25.136105 (+ 40us) server_negotiation.cc:407] Connection header received
0619 16:53:25.136262 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:25.136271 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:25.136341 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:25.136498 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:25.137287 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:25.138781 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:25.139940 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:25.140601 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:25.140709 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:25.143996 (+ 3287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:25.144003 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:25.144003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:25.144021 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:25.146372 (+ 2351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:25.146890 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:25.146894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:25.146895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:25.147031 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:25.147238 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:25.147239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:25.147239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:25.147617 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:25.147804 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:25.147891 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 16:53:25.148001 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
I20260619 16:53:25.149380 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:53:25.150969 20705 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:53:25.151757 20705 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:53:25.155776 20705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09a76dcc66074a79b4f886fe359cb178: Generated new TSK 0
I20260619 16:53:25.156049 20705 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:53:25.158509 20715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:25.151083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34025 (local address 127.0.0.1:46041)
0619 16:53:25.151121 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:25.151122 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:25.151269 (+ 147us) server_negotiation.cc:407] Connection header received
0619 16:53:25.151332 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:25.151333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:25.151344 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:25.151408 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:25.152045 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:25.152863 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:25.153854 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:25.154176 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:25.154277 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:25.155870 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:25.155872 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:25.155872 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:25.155880 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:25.157340 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:25.157751 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:25.157752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:25.157752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:25.157799 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:25.157990 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:25.157991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:25.157991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:25.158094 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:25.158174 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:25.158368 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 16:53:25.158423 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 16:53:25.160926 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 20626
W20260619 16:53:25.165433 20611 connection.cc:535] client connection to 127.0.0.1:46041 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46041): Connection reset by peer (error 104)
2026-06-19T16:53:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1287 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:53:25 dist-test-slave-jsw9 krb5kdc[20725](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:53:25 dist-test-slave-jsw9 krb5kdc[20725](info): set up 2 sockets
Jun 19 16:53:25 dist-test-slave-jsw9 krb5kdc[20725](info): commencing operation
krb5kdc: starting...
W20260619 16:53:26.138549 392 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:53:26 dist-test-slave-jsw9 krb5kdc[20725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:53:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:53:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:53:26.269171 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38611
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:53:26.377350 20738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:26.377622 20738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:26.377660 20738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:26.379070 20738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:53:26.379130 20738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:26.379150 20738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:26.379166 20738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:53:26.379182 20738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:53:26.380872 20738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38611
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:26.381179 20738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:26.381435 20738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:26.387599 20744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:26.387683 20738 server_base.cc:1034] running on GCE node
W20260619 16:53:26.387598 20746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:26.387598 20743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:26.388041 20738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:26.388362 20738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:26.389524 20738 hybrid_clock.cc:648] HybridClock initialized: now 1781888006389508 us; error 29 us; skew 500 ppm
Jun 19 16:53:26 dist-test-slave-jsw9 krb5kdc[20725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:26.392398 20738 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:53:26.392910 20738 webserver.cc:460] Webserver started at http://127.0.0.1:33613/ using document root <none> and password file <none>
I20260619 16:53:26.393142 20738 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:26.393193 20738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:26.393347 20738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:26.394618 20738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "9bd4055a7b6d4ff2930452e1a9743a37"
format_stamp: "Formatted at 2026-06-19 16:53:26 on dist-test-slave-jsw9"
server_key: "86a42e982fedf6e6d5778bfbde282782"
server_key_iv: "01f063b8bf3220ceb600cde84f3d4fea"
server_key_version: "encryptionkey@0"
I20260619 16:53:26.394989 20738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "9bd4055a7b6d4ff2930452e1a9743a37"
format_stamp: "Formatted at 2026-06-19 16:53:26 on dist-test-slave-jsw9"
server_key: "86a42e982fedf6e6d5778bfbde282782"
server_key_iv: "01f063b8bf3220ceb600cde84f3d4fea"
server_key_version: "encryptionkey@0"
I20260619 16:53:26.396519 20738 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 16:53:26.397379 20753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:26.397564 20738 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:53:26.397658 20738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "9bd4055a7b6d4ff2930452e1a9743a37"
format_stamp: "Formatted at 2026-06-19 16:53:26 on dist-test-slave-jsw9"
server_key: "86a42e982fedf6e6d5778bfbde282782"
server_key_iv: "01f063b8bf3220ceb600cde84f3d4fea"
server_key_version: "encryptionkey@0"
I20260619 16:53:26.397729 20738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:26.410115 20738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:26.412582 20738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:26.412750 20738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:26.418305 20738 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38611
I20260619 16:53:26.418354 20805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38611 every 8 connection(s)
I20260619 16:53:26.418746 20738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:53:26.419814 20806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:26.422595 20806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bd4055a7b6d4ff2930452e1a9743a37: Bootstrap starting.
I20260619 16:53:26.423290 20806 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bd4055a7b6d4ff2930452e1a9743a37: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:26.423584 20806 log.cc:824] T 00000000000000000000000000000000 P 9bd4055a7b6d4ff2930452e1a9743a37: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:26.424386 20806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bd4055a7b6d4ff2930452e1a9743a37: No bootstrap required, opened a new log
I20260619 16:53:26.426009 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 20738
I20260619 16:53:26.426115 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:53:26.426362 392 external_mini_cluster.cc:1428] Setting key ac8e04b205c7dcccff5da1d1f4020da8
I20260619 16:53:26.426235 20806 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bd4055a7b6d4ff2930452e1a9743a37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd4055a7b6d4ff2930452e1a9743a37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38611 } }
I20260619 16:53:26.426394 20806 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bd4055a7b6d4ff2930452e1a9743a37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:26.426427 20806 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bd4055a7b6d4ff2930452e1a9743a37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bd4055a7b6d4ff2930452e1a9743a37, State: Initialized, Role: FOLLOWER
I20260619 16:53:26.426596 20806 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bd4055a7b6d4ff2930452e1a9743a37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd4055a7b6d4ff2930452e1a9743a37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38611 } }
I20260619 16:53:26.426685 20806 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bd4055a7b6d4ff2930452e1a9743a37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:53:26.426716 20806 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bd4055a7b6d4ff2930452e1a9743a37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:53:26.426769 20806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bd4055a7b6d4ff2930452e1a9743a37 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:26.427968 20806 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bd4055a7b6d4ff2930452e1a9743a37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd4055a7b6d4ff2930452e1a9743a37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38611 } }
I20260619 16:53:26.428085 20806 leader_election.cc:302] T 00000000000000000000000000000000 P 9bd4055a7b6d4ff2930452e1a9743a37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bd4055a7b6d4ff2930452e1a9743a37; no voters:
I20260619 16:53:26.428267 20806 leader_election.cc:288] T 00000000000000000000000000000000 P 9bd4055a7b6d4ff2930452e1a9743a37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:53:26.428320 20811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bd4055a7b6d4ff2930452e1a9743a37 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:53:26.428465 20811 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bd4055a7b6d4ff2930452e1a9743a37 [term 1 LEADER]: Becoming Leader. State: Replica: 9bd4055a7b6d4ff2930452e1a9743a37, State: Running, Role: LEADER
I20260619 16:53:26.428542 20806 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bd4055a7b6d4ff2930452e1a9743a37 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:53:26.428579 20811 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bd4055a7b6d4ff2930452e1a9743a37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd4055a7b6d4ff2930452e1a9743a37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38611 } }
I20260619 16:53:26.428972 20811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bd4055a7b6d4ff2930452e1a9743a37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bd4055a7b6d4ff2930452e1a9743a37. Latest consensus state: current_term: 1 leader_uuid: "9bd4055a7b6d4ff2930452e1a9743a37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd4055a7b6d4ff2930452e1a9743a37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38611 } } }
I20260619 16:53:26.429199 20811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bd4055a7b6d4ff2930452e1a9743a37 [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:26.429111 20812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bd4055a7b6d4ff2930452e1a9743a37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bd4055a7b6d4ff2930452e1a9743a37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd4055a7b6d4ff2930452e1a9743a37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38611 } } }
I20260619 16:53:26.429613 20812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bd4055a7b6d4ff2930452e1a9743a37 [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:26.429877 20820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:53:26.430409 20820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:53:26.432364 20820 catalog_manager.cc:1347] Generated new cluster ID: 275d775794f6457cb3a8afcf06144fbe
I20260619 16:53:26.432466 20820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:53:26.432731 20810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:26.426899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50733 (local address 127.0.0.1:38611)
0619 16:53:26.427132 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:26.427143 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:26.427182 (+ 39us) server_negotiation.cc:407] Connection header received
0619 16:53:26.427325 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:26.427333 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:26.427398 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:26.427585 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:26.427728 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:26.429127 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:26.430377 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:26.430985 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:26.431485 (+ 500us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:26.431918 (+ 433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:26.431922 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:26.431922 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:26.432182 (+ 260us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:26.432229 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:26.432243 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:26.432316 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:26.432469 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 16:53:26.432504 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":125,"threads_started":1}
I20260619 16:53:26.433735 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:53:26.442307 20810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:26.435409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35001 (local address 127.0.0.1:38611)
0619 16:53:26.435445 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:26.435447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:26.438380 (+ 2933us) server_negotiation.cc:407] Connection header received
0619 16:53:26.438487 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:26.438489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:26.438504 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:26.438575 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:26.438731 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:26.439702 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:26.440757 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:26.441115 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:26.441186 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:26.441989 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:26.441991 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:26.441991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:26.441997 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:26.442017 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:26.442022 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:26.442086 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:26.442143 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 16:53:26.442160 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 16:53:26.451092 20820 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:53:26.451860 20820 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:53:26.457464 20820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bd4055a7b6d4ff2930452e1a9743a37: Generated new TSK 0
I20260619 16:53:26.457692 20820 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:53:26.465873 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 20738
W20260619 16:53:26.470865 20723 connection.cc:535] client connection to 127.0.0.1:38611 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38611): Connection reset by peer (error 104)
2026-06-19T16:53:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1305 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:53:26 dist-test-slave-jsw9 krb5kdc[20837](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:53:26 dist-test-slave-jsw9 krb5kdc[20837](info): set up 2 sockets
Jun 19 16:53:26 dist-test-slave-jsw9 krb5kdc[20837](info): commencing operation
krb5kdc: starting...
W20260619 16:53:27.426759 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:53:27 dist-test-slave-jsw9 krb5kdc[20837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:53:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:53:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:53:27.561429 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38965
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:53:27.658277 20850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:27.658563 20850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:27.658602 20850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:27.660143 20850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:53:27.660216 20850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:27.660238 20850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:27.660256 20850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:53:27.660271 20850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:53:27.662250 20850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38965
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:27.662581 20850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:27.662859 20850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:27.668519 20856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:27.668519 20858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:27.668519 20855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:27.668924 20850 server_base.cc:1034] running on GCE node
I20260619 16:53:27.669364 20850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:27.669751 20850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:27.670933 20850 hybrid_clock.cc:648] HybridClock initialized: now 1781888007670894 us; error 59 us; skew 500 ppm
Jun 19 16:53:27 dist-test-slave-jsw9 krb5kdc[20837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:27.674778 20850 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:53:27.675431 20850 webserver.cc:460] Webserver started at http://127.0.0.1:41627/ using document root <none> and password file <none>
I20260619 16:53:27.675688 20850 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:27.675742 20850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:27.675887 20850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:27.677347 20850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "5a857091819648b787723f23cd4366eb"
format_stamp: "Formatted at 2026-06-19 16:53:27 on dist-test-slave-jsw9"
server_key: "3611ee2c35a1e384c4cc83b3cea5d172"
server_key_iv: "f05b0ba248b48000b8407b8306ec30fb"
server_key_version: "encryptionkey@0"
I20260619 16:53:27.677745 20850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "5a857091819648b787723f23cd4366eb"
format_stamp: "Formatted at 2026-06-19 16:53:27 on dist-test-slave-jsw9"
server_key: "3611ee2c35a1e384c4cc83b3cea5d172"
server_key_iv: "f05b0ba248b48000b8407b8306ec30fb"
server_key_version: "encryptionkey@0"
I20260619 16:53:27.679394 20850 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260619 16:53:27.680346 20865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:27.680620 20850 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:53:27.680732 20850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "5a857091819648b787723f23cd4366eb"
format_stamp: "Formatted at 2026-06-19 16:53:27 on dist-test-slave-jsw9"
server_key: "3611ee2c35a1e384c4cc83b3cea5d172"
server_key_iv: "f05b0ba248b48000b8407b8306ec30fb"
server_key_version: "encryptionkey@0"
I20260619 16:53:27.680814 20850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:27.684868 20850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:27.687381 20850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:27.687557 20850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:27.693272 20850 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38965
I20260619 16:53:27.693310 20917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38965 every 8 connection(s)
I20260619 16:53:27.693724 20850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:53:27.694926 20918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:27.697829 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 20850
I20260619 16:53:27.698005 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:53:27.698119 20918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a857091819648b787723f23cd4366eb: Bootstrap starting.
I20260619 16:53:27.698233 392 external_mini_cluster.cc:1428] Setting key 1c3bc4061f8bc9aeeee6a999e48ffb58
I20260619 16:53:27.698992 20918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a857091819648b787723f23cd4366eb: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:27.699438 20918 log.cc:824] T 00000000000000000000000000000000 P 5a857091819648b787723f23cd4366eb: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:27.700263 20918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a857091819648b787723f23cd4366eb: No bootstrap required, opened a new log
I20260619 16:53:27.702486 20918 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a857091819648b787723f23cd4366eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a857091819648b787723f23cd4366eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38965 } }
I20260619 16:53:27.702702 20918 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a857091819648b787723f23cd4366eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:27.702737 20918 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a857091819648b787723f23cd4366eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a857091819648b787723f23cd4366eb, State: Initialized, Role: FOLLOWER
I20260619 16:53:27.702957 20921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:27.698884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34895 (local address 127.0.0.1:38965)
0619 16:53:27.699105 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:27.699116 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:27.699153 (+ 37us) server_negotiation.cc:407] Connection header received
0619 16:53:27.699843 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:27.699852 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:27.699885 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:27.700167 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:27.702128 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:27.702134 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:27.702135 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:27.702141 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:27.702239 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:27.702260 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:27.702333 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:27.702475 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 16:53:27.702528 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":92,"threads_started":1}
I20260619 16:53:27.702917 20918 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a857091819648b787723f23cd4366eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a857091819648b787723f23cd4366eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38965 } }
I20260619 16:53:27.703018 20918 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a857091819648b787723f23cd4366eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:53:27.703052 20918 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a857091819648b787723f23cd4366eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:53:27.703095 20918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a857091819648b787723f23cd4366eb [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:27.704023 20918 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a857091819648b787723f23cd4366eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a857091819648b787723f23cd4366eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38965 } }
I20260619 16:53:27.704149 20918 leader_election.cc:302] T 00000000000000000000000000000000 P 5a857091819648b787723f23cd4366eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a857091819648b787723f23cd4366eb; no voters:
I20260619 16:53:27.704249 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:53:27.704356 20918 leader_election.cc:288] T 00000000000000000000000000000000 P 5a857091819648b787723f23cd4366eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:53:27.704411 20923 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a857091819648b787723f23cd4366eb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:53:27.704564 20923 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a857091819648b787723f23cd4366eb [term 1 LEADER]: Becoming Leader. State: Replica: 5a857091819648b787723f23cd4366eb, State: Running, Role: LEADER
I20260619 16:53:27.704658 20918 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a857091819648b787723f23cd4366eb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:53:27.704738 20923 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a857091819648b787723f23cd4366eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a857091819648b787723f23cd4366eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38965 } }
I20260619 16:53:27.705135 20923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a857091819648b787723f23cd4366eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a857091819648b787723f23cd4366eb. Latest consensus state: current_term: 1 leader_uuid: "5a857091819648b787723f23cd4366eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a857091819648b787723f23cd4366eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38965 } } }
I20260619 16:53:27.705267 20923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a857091819648b787723f23cd4366eb [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:27.705969 20932 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:53:27.706233 20925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a857091819648b787723f23cd4366eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a857091819648b787723f23cd4366eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a857091819648b787723f23cd4366eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38965 } } }
I20260619 16:53:27.706537 20925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a857091819648b787723f23cd4366eb [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:27.706629 20932 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:53:27.707819 20921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:27.706085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45331 (local address 127.0.0.1:38965)
0619 16:53:27.706126 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:27.706128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:27.706556 (+ 428us) server_negotiation.cc:407] Connection header received
0619 16:53:27.706660 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:27.706662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:27.706668 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:27.706698 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:27.707611 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:27.707612 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:27.707613 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:27.707617 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:27.707636 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:27.707641 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:27.707697 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:27.707741 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 16:53:27.707756 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 16:53:27.711638 20932 catalog_manager.cc:1347] Generated new cluster ID: 8733c54f6c824c579633878b3f47bd48
I20260619 16:53:27.711724 20932 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:53:27.736148 20932 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:53:27.736402 20932 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:53:27.743781 20932 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a857091819648b787723f23cd4366eb: Generated new TSK 0
I20260619 16:53:27.744086 20932 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:53:27.745913 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 20850
2026-06-19T16:53:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1280 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:53:27 dist-test-slave-jsw9 krb5kdc[20949](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:53:27 dist-test-slave-jsw9 krb5kdc[20949](info): set up 2 sockets
Jun 19 16:53:27 dist-test-slave-jsw9 krb5kdc[20949](info): commencing operation
krb5kdc: starting...
W20260619 16:53:28.730440 392 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:53:28 dist-test-slave-jsw9 krb5kdc[20949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:53:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:53:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:53:28.857599 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44391
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:53:28.946215 20962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:28.946482 20962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:28.946518 20962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:28.947860 20962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:53:28.947919 20962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:28.947938 20962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:28.947954 20962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:53:28.947969 20962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:53:28.949652 20962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:28.949977 20962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:28.950260 20962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:28.955626 20968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:28.955580 20970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:28.955580 20967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:28.956291 20962 server_base.cc:1034] running on GCE node
I20260619 16:53:28.956670 20962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:28.957051 20962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:28.958259 20962 hybrid_clock.cc:648] HybridClock initialized: now 1781888008958211 us; error 66 us; skew 500 ppm
Jun 19 16:53:28 dist-test-slave-jsw9 krb5kdc[20949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:28.961297 20962 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:53:28.961849 20962 webserver.cc:460] Webserver started at http://127.0.0.1:33453/ using document root <none> and password file <none>
I20260619 16:53:28.962080 20962 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:28.962131 20962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:28.962303 20962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:28.963588 20962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "e9c1124efcc44c20bd86bfa260a443c8"
format_stamp: "Formatted at 2026-06-19 16:53:28 on dist-test-slave-jsw9"
server_key: "4ff8d14bab081a4ed9769020f208541c"
server_key_iv: "dc136a0f4266fa88ae5a80ccccdeaa35"
server_key_version: "encryptionkey@0"
I20260619 16:53:28.963977 20962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "e9c1124efcc44c20bd86bfa260a443c8"
format_stamp: "Formatted at 2026-06-19 16:53:28 on dist-test-slave-jsw9"
server_key: "4ff8d14bab081a4ed9769020f208541c"
server_key_iv: "dc136a0f4266fa88ae5a80ccccdeaa35"
server_key_version: "encryptionkey@0"
I20260619 16:53:28.965660 20962 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 16:53:28.966604 20977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:28.966771 20962 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:53:28.966885 20962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "e9c1124efcc44c20bd86bfa260a443c8"
format_stamp: "Formatted at 2026-06-19 16:53:28 on dist-test-slave-jsw9"
server_key: "4ff8d14bab081a4ed9769020f208541c"
server_key_iv: "dc136a0f4266fa88ae5a80ccccdeaa35"
server_key_version: "encryptionkey@0"
I20260619 16:53:28.966956 20962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:28.990011 20962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:28.992463 20962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:28.992624 20962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:28.997288 20962 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44391
I20260619 16:53:28.997344 21029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44391 every 8 connection(s)
I20260619 16:53:28.997740 20962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:53:28.998818 21030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:29.001596 21030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9c1124efcc44c20bd86bfa260a443c8: Bootstrap starting.
I20260619 16:53:29.002285 21030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9c1124efcc44c20bd86bfa260a443c8: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:29.002559 21030 log.cc:824] T 00000000000000000000000000000000 P e9c1124efcc44c20bd86bfa260a443c8: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:29.003388 21030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9c1124efcc44c20bd86bfa260a443c8: No bootstrap required, opened a new log
I20260619 16:53:29.003836 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 20962
I20260619 16:53:29.003976 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:53:29.004164 392 external_mini_cluster.cc:1428] Setting key 65d2fb6181223064f35cba0ad8227e36
I20260619 16:53:29.005445 21030 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9c1124efcc44c20bd86bfa260a443c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9c1124efcc44c20bd86bfa260a443c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44391 } }
I20260619 16:53:29.005626 21030 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9c1124efcc44c20bd86bfa260a443c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:29.005650 21030 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9c1124efcc44c20bd86bfa260a443c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9c1124efcc44c20bd86bfa260a443c8, State: Initialized, Role: FOLLOWER
I20260619 16:53:29.005791 21030 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9c1124efcc44c20bd86bfa260a443c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9c1124efcc44c20bd86bfa260a443c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44391 } }
I20260619 16:53:29.005887 21030 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9c1124efcc44c20bd86bfa260a443c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:53:29.005918 21030 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9c1124efcc44c20bd86bfa260a443c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:53:29.005973 21030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9c1124efcc44c20bd86bfa260a443c8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:29.007822 21030 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9c1124efcc44c20bd86bfa260a443c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9c1124efcc44c20bd86bfa260a443c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44391 } }
I20260619 16:53:29.007992 21030 leader_election.cc:302] T 00000000000000000000000000000000 P e9c1124efcc44c20bd86bfa260a443c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9c1124efcc44c20bd86bfa260a443c8; no voters:
I20260619 16:53:29.008231 21030 leader_election.cc:288] T 00000000000000000000000000000000 P e9c1124efcc44c20bd86bfa260a443c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:53:29.008272 21035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9c1124efcc44c20bd86bfa260a443c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:53:29.008432 21035 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9c1124efcc44c20bd86bfa260a443c8 [term 1 LEADER]: Becoming Leader. State: Replica: e9c1124efcc44c20bd86bfa260a443c8, State: Running, Role: LEADER
I20260619 16:53:29.008581 21035 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9c1124efcc44c20bd86bfa260a443c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9c1124efcc44c20bd86bfa260a443c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44391 } }
I20260619 16:53:29.008627 21030 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9c1124efcc44c20bd86bfa260a443c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:53:29.008981 21035 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9c1124efcc44c20bd86bfa260a443c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9c1124efcc44c20bd86bfa260a443c8. Latest consensus state: current_term: 1 leader_uuid: "e9c1124efcc44c20bd86bfa260a443c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9c1124efcc44c20bd86bfa260a443c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44391 } } }
I20260619 16:53:29.009032 21037 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9c1124efcc44c20bd86bfa260a443c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9c1124efcc44c20bd86bfa260a443c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9c1124efcc44c20bd86bfa260a443c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44391 } } }
I20260619 16:53:29.009124 21035 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9c1124efcc44c20bd86bfa260a443c8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:29.009130 21037 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9c1124efcc44c20bd86bfa260a443c8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:29.009373 21042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:53:29.010025 21042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:53:29.012158 21042 catalog_manager.cc:1347] Generated new cluster ID: edb35a06d38b4350901f269ef70904b1
I20260619 16:53:29.012220 21042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 16:53:29 dist-test-slave-jsw9 krb5kdc[20949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 16:53:29.019768 21034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:29.004759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50043 (local address 127.0.0.1:44391)
0619 16:53:29.004983 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:29.004994 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:29.006018 (+ 1024us) server_negotiation.cc:407] Connection header received
0619 16:53:29.006890 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:29.006907 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:29.007094 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:29.007387 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:29.008541 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:29.010050 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:29.011320 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:29.011860 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:29.011984 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:29.015433 (+ 3449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:29.015441 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:29.015441 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:29.015462 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:29.017765 (+ 2303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:29.018240 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:29.018245 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:29.018246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:29.018395 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:29.018591 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:29.018591 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:29.018592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:29.019009 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:29.019197 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:29.019289 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 16:53:29.019476 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":104,"threads_started":1}
I20260619 16:53:29.020996 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:53:29.034569 21034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:29.022785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44623 (local address 127.0.0.1:44391)
0619 16:53:29.022829 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:29.022831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:29.026464 (+ 3633us) server_negotiation.cc:407] Connection header received
0619 16:53:29.026598 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:29.026600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:29.026616 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:29.026685 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:29.027465 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:29.028316 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:29.029430 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:29.029749 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:29.029810 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:29.031553 (+ 1743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:29.031555 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:29.031556 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:29.031563 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:29.033258 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:29.033697 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:29.033698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:29.033699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:29.033761 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:29.033950 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:29.033951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:29.033951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:29.034089 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:29.034272 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:29.034353 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 16:53:29.034453 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 16:53:29.039682 21042 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:53:29.040269 21042 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:53:29.044453 21042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9c1124efcc44c20bd86bfa260a443c8: Generated new TSK 0
I20260619 16:53:29.044760 21042 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:53:29.052596 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 20962
2026-06-19T16:53:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1305 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:53:29 dist-test-slave-jsw9 krb5kdc[21061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:53:29 dist-test-slave-jsw9 krb5kdc[21061](info): set up 2 sockets
Jun 19 16:53:29 dist-test-slave-jsw9 krb5kdc[21061](info): commencing operation
krb5kdc: starting...
W20260619 16:53:30.014540 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:53:30 dist-test-slave-jsw9 krb5kdc[21061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:53:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:53:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:53:30.149950 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42899
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:53:30.243613 21074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:30.243920 21074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:30.243958 21074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:30.245476 21074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:53:30.245548 21074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:30.245569 21074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:30.245587 21074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:53:30.245604 21074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:53:30.247637 21074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:30.247983 21074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:30.248255 21074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:30.254297 21079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:30.254319 21082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:30.254504 21074 server_base.cc:1034] running on GCE node
W20260619 16:53:30.254302 21080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:30.254866 21074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:30.255245 21074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:30.256394 21074 hybrid_clock.cc:648] HybridClock initialized: now 1781888010256395 us; error 51 us; skew 500 ppm
Jun 19 16:53:30 dist-test-slave-jsw9 krb5kdc[21061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:30.262514 21074 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:53:30.263116 21074 webserver.cc:460] Webserver started at http://127.0.0.1:44749/ using document root <none> and password file <none>
I20260619 16:53:30.263389 21074 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:30.263445 21074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:30.263588 21074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:30.264828 21074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "1f96cffa4504406fbe66daad4adccba7"
format_stamp: "Formatted at 2026-06-19 16:53:30 on dist-test-slave-jsw9"
server_key: "1df3e4d92a20ad7f177d8d73d8a22a6d"
server_key_iv: "ce38a3e826ef4cc5d35728b2496bfe4a"
server_key_version: "encryptionkey@0"
I20260619 16:53:30.265205 21074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "1f96cffa4504406fbe66daad4adccba7"
format_stamp: "Formatted at 2026-06-19 16:53:30 on dist-test-slave-jsw9"
server_key: "1df3e4d92a20ad7f177d8d73d8a22a6d"
server_key_iv: "ce38a3e826ef4cc5d35728b2496bfe4a"
server_key_version: "encryptionkey@0"
I20260619 16:53:30.266749 21074 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 16:53:30.267633 21089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:30.267874 21074 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:53:30.267984 21074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "1f96cffa4504406fbe66daad4adccba7"
format_stamp: "Formatted at 2026-06-19 16:53:30 on dist-test-slave-jsw9"
server_key: "1df3e4d92a20ad7f177d8d73d8a22a6d"
server_key_iv: "ce38a3e826ef4cc5d35728b2496bfe4a"
server_key_version: "encryptionkey@0"
I20260619 16:53:30.268056 21074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:30.294052 21074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:30.297206 21074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:30.297413 21074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:30.302591 21074 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42899
I20260619 16:53:30.302670 21141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42899 every 8 connection(s)
I20260619 16:53:30.303057 21074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:53:30.304364 21142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:30.306758 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 21074
I20260619 16:53:30.306891 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:53:30.307099 392 external_mini_cluster.cc:1428] Setting key 37d9cef3000a87553d57a759f2880047
I20260619 16:53:30.307907 21142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f96cffa4504406fbe66daad4adccba7: Bootstrap starting.
I20260619 16:53:30.308699 21142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f96cffa4504406fbe66daad4adccba7: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:30.309015 21142 log.cc:824] T 00000000000000000000000000000000 P 1f96cffa4504406fbe66daad4adccba7: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:30.309885 21142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f96cffa4504406fbe66daad4adccba7: No bootstrap required, opened a new log
I20260619 16:53:30.312412 21142 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f96cffa4504406fbe66daad4adccba7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f96cffa4504406fbe66daad4adccba7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42899 } }
I20260619 16:53:30.312638 21142 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f96cffa4504406fbe66daad4adccba7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:30.312683 21142 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f96cffa4504406fbe66daad4adccba7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f96cffa4504406fbe66daad4adccba7, State: Initialized, Role: FOLLOWER
I20260619 16:53:30.313151 21142 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f96cffa4504406fbe66daad4adccba7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f96cffa4504406fbe66daad4adccba7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42899 } }
I20260619 16:53:30.313279 21142 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f96cffa4504406fbe66daad4adccba7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:53:30.313325 21142 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f96cffa4504406fbe66daad4adccba7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:53:30.313380 21142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f96cffa4504406fbe66daad4adccba7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:30.314277 21145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:30.307796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45087 (local address 127.0.0.1:42899)
0619 16:53:30.308019 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:30.308030 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:30.308070 (+ 40us) server_negotiation.cc:407] Connection header received
0619 16:53:30.308631 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:30.308640 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:30.308704 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:30.308865 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:30.309181 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:30.310583 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:30.311538 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:30.312309 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:30.312441 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:30.313242 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:30.313247 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:30.313248 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:30.313509 (+ 261us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:30.313559 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:30.313578 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:30.313655 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:30.313900 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 16:53:30.313996 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":111,"threads_started":1}
I20260619 16:53:30.314337 21142 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f96cffa4504406fbe66daad4adccba7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f96cffa4504406fbe66daad4adccba7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42899 } }
I20260619 16:53:30.314443 21142 leader_election.cc:302] T 00000000000000000000000000000000 P 1f96cffa4504406fbe66daad4adccba7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f96cffa4504406fbe66daad4adccba7; no voters:
I20260619 16:53:30.314661 21142 leader_election.cc:288] T 00000000000000000000000000000000 P 1f96cffa4504406fbe66daad4adccba7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:53:30.314719 21147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f96cffa4504406fbe66daad4adccba7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:53:30.314888 21147 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f96cffa4504406fbe66daad4adccba7 [term 1 LEADER]: Becoming Leader. State: Replica: 1f96cffa4504406fbe66daad4adccba7, State: Running, Role: LEADER
I20260619 16:53:30.315021 21142 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f96cffa4504406fbe66daad4adccba7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:53:30.315181 21147 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f96cffa4504406fbe66daad4adccba7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f96cffa4504406fbe66daad4adccba7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42899 } }
I20260619 16:53:30.315414 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:53:30.316900 21149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f96cffa4504406fbe66daad4adccba7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f96cffa4504406fbe66daad4adccba7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f96cffa4504406fbe66daad4adccba7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42899 } } }
I20260619 16:53:30.317132 21149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f96cffa4504406fbe66daad4adccba7 [sys.catalog]: This master's current role is: LEADER
W20260619 16:53:30.317229 21162 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1f96cffa4504406fbe66daad4adccba7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260619 16:53:30.317354 21148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f96cffa4504406fbe66daad4adccba7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f96cffa4504406fbe66daad4adccba7. Latest consensus state: current_term: 1 leader_uuid: "1f96cffa4504406fbe66daad4adccba7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f96cffa4504406fbe66daad4adccba7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42899 } } }
I20260619 16:53:30.317471 21148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f96cffa4504406fbe66daad4adccba7 [sys.catalog]: This master's current role is: LEADER
W20260619 16:53:30.317400 21162 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:53:30.317898 21168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:53:30.318297 21168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:53:30.320535 21168 catalog_manager.cc:1347] Generated new cluster ID: 323f595cc80a46aaa4e9720e771b3939
I20260619 16:53:30.320600 21168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:53:30.321560 21145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:30.317219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54041 (local address 127.0.0.1:42899)
0619 16:53:30.317265 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:30.317267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:30.317672 (+ 405us) server_negotiation.cc:407] Connection header received
0619 16:53:30.317727 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:30.317729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:30.317742 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:30.317882 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:30.318296 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:30.319039 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:30.320043 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:30.320323 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:30.320391 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:30.321222 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:30.321224 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:30.321224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:30.321230 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:30.321248 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:30.321253 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:30.321290 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:30.321456 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 16:53:30.321482 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 16:53:30.335002 21168 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:53:30.335938 21168 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:53:30.341114 21168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f96cffa4504406fbe66daad4adccba7: Generated new TSK 0
I20260619 16:53:30.341409 21168 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:53:30.348227 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 21074
2026-06-19T16:53:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1296 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:53:30 dist-test-slave-jsw9 krb5kdc[21173](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:53:30 dist-test-slave-jsw9 krb5kdc[21173](info): set up 2 sockets
Jun 19 16:53:30 dist-test-slave-jsw9 krb5kdc[21173](info): commencing operation
krb5kdc: starting...
W20260619 16:53:31.322505 392 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:53:31 dist-test-slave-jsw9 krb5kdc[21173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:53:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:53:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:53:31.454515 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38679
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:53:31.545614 21186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:31.545930 21186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:31.545969 21186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:31.547363 21186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:53:31.547418 21186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:31.547437 21186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:31.547454 21186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:53:31.547468 21186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:53:31.549180 21186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:31.549472 21186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:31.549731 21186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:31.555152 21194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:31.555154 21191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:31.555154 21192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:31.555387 21186 server_base.cc:1034] running on GCE node
I20260619 16:53:31.555639 21186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:31.555951 21186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:31.557132 21186 hybrid_clock.cc:648] HybridClock initialized: now 1781888011557104 us; error 54 us; skew 500 ppm
Jun 19 16:53:31 dist-test-slave-jsw9 krb5kdc[21173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:31.560060 21186 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:53:31.560547 21186 webserver.cc:460] Webserver started at http://127.0.0.1:36797/ using document root <none> and password file <none>
I20260619 16:53:31.560745 21186 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:31.560784 21186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:31.560897 21186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:31.562076 21186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "f806e92acab647c6892e71149dd73dcf"
format_stamp: "Formatted at 2026-06-19 16:53:31 on dist-test-slave-jsw9"
server_key: "8c1dd1043eaeb3ce2ea30186b6dba96f"
server_key_iv: "8cb11e7e45ebeff6bb0c0070ddf8d724"
server_key_version: "encryptionkey@0"
I20260619 16:53:31.562496 21186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "f806e92acab647c6892e71149dd73dcf"
format_stamp: "Formatted at 2026-06-19 16:53:31 on dist-test-slave-jsw9"
server_key: "8c1dd1043eaeb3ce2ea30186b6dba96f"
server_key_iv: "8cb11e7e45ebeff6bb0c0070ddf8d724"
server_key_version: "encryptionkey@0"
I20260619 16:53:31.563918 21186 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:53:31.564664 21201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:31.564838 21186 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:53:31.564949 21186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "f806e92acab647c6892e71149dd73dcf"
format_stamp: "Formatted at 2026-06-19 16:53:31 on dist-test-slave-jsw9"
server_key: "8c1dd1043eaeb3ce2ea30186b6dba96f"
server_key_iv: "8cb11e7e45ebeff6bb0c0070ddf8d724"
server_key_version: "encryptionkey@0"
I20260619 16:53:31.565016 21186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:31.568768 21186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:31.571265 21186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:31.571421 21186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:31.576099 21186 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38679
I20260619 16:53:31.576143 21253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38679 every 8 connection(s)
I20260619 16:53:31.576525 21186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:53:31.577630 21254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:31.580154 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 21186
I20260619 16:53:31.580297 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:53:31.580513 392 external_mini_cluster.cc:1428] Setting key a637fb2e148499e404892bac9cf18345
I20260619 16:53:31.580927 21254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f806e92acab647c6892e71149dd73dcf: Bootstrap starting.
I20260619 16:53:31.581688 21254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f806e92acab647c6892e71149dd73dcf: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:31.581987 21254 log.cc:824] T 00000000000000000000000000000000 P f806e92acab647c6892e71149dd73dcf: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:31.582823 21254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f806e92acab647c6892e71149dd73dcf: No bootstrap required, opened a new log
I20260619 16:53:31.583307 21257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:31.581089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36615 (local address 127.0.0.1:38679)
0619 16:53:31.581299 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:31.581310 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:31.581352 (+ 42us) server_negotiation.cc:407] Connection header received
0619 16:53:31.581916 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:31.581924 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:31.581956 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:31.582105 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:31.582849 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:31.582853 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:31.582854 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:31.582857 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:31.582911 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:31.582928 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:31.583034 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:31.583067 (+ 33us) server_negotiation.cc:299] Negotiation successful
0619 16:53:31.583113 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":96,"threads_started":1}
I20260619 16:53:31.584359 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:53:31.585248 21254 raft_consensus.cc:348] T 00000000000000000000000000000000 P f806e92acab647c6892e71149dd73dcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f806e92acab647c6892e71149dd73dcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38679 } }
I20260619 16:53:31.585419 21254 raft_consensus.cc:374] T 00000000000000000000000000000000 P f806e92acab647c6892e71149dd73dcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:31.585464 21254 raft_consensus.cc:729] T 00000000000000000000000000000000 P f806e92acab647c6892e71149dd73dcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f806e92acab647c6892e71149dd73dcf, State: Initialized, Role: FOLLOWER
I20260619 16:53:31.585605 21254 consensus_queue.cc:260] T 00000000000000000000000000000000 P f806e92acab647c6892e71149dd73dcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f806e92acab647c6892e71149dd73dcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38679 } }
I20260619 16:53:31.585705 21254 raft_consensus.cc:388] T 00000000000000000000000000000000 P f806e92acab647c6892e71149dd73dcf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:53:31.585747 21254 raft_consensus.cc:482] T 00000000000000000000000000000000 P f806e92acab647c6892e71149dd73dcf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:53:31.585798 21254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f806e92acab647c6892e71149dd73dcf [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:31.586584 21254 raft_consensus.cc:504] T 00000000000000000000000000000000 P f806e92acab647c6892e71149dd73dcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f806e92acab647c6892e71149dd73dcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38679 } }
I20260619 16:53:31.586706 21254 leader_election.cc:302] T 00000000000000000000000000000000 P f806e92acab647c6892e71149dd73dcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f806e92acab647c6892e71149dd73dcf; no voters:
I20260619 16:53:31.586911 21254 leader_election.cc:288] T 00000000000000000000000000000000 P f806e92acab647c6892e71149dd73dcf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:53:31.586990 21257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:31.585849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37145 (local address 127.0.0.1:38679)
0619 16:53:31.585887 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:31.585889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:31.586020 (+ 131us) server_negotiation.cc:407] Connection header received
0619 16:53:31.586077 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:31.586078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:31.586083 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:31.586138 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:31.586779 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:31.586780 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:31.586780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:31.586785 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:31.586801 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:31.586805 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:31.586867 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:31.586912 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 16:53:31.586926 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 16:53:31.587056 21265 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f806e92acab647c6892e71149dd73dcf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:53:31.587172 21265 raft_consensus.cc:686] T 00000000000000000000000000000000 P f806e92acab647c6892e71149dd73dcf [term 1 LEADER]: Becoming Leader. State: Replica: f806e92acab647c6892e71149dd73dcf, State: Running, Role: LEADER
I20260619 16:53:31.587211 21254 sys_catalog.cc:564] T 00000000000000000000000000000000 P f806e92acab647c6892e71149dd73dcf [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:53:31.587276 21265 consensus_queue.cc:237] T 00000000000000000000000000000000 P f806e92acab647c6892e71149dd73dcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f806e92acab647c6892e71149dd73dcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38679 } }
I20260619 16:53:31.587702 21264 sys_catalog.cc:455] T 00000000000000000000000000000000 P f806e92acab647c6892e71149dd73dcf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f806e92acab647c6892e71149dd73dcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f806e92acab647c6892e71149dd73dcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38679 } } }
I20260619 16:53:31.587716 21266 sys_catalog.cc:455] T 00000000000000000000000000000000 P f806e92acab647c6892e71149dd73dcf [sys.catalog]: SysCatalogTable state changed. Reason: New leader f806e92acab647c6892e71149dd73dcf. Latest consensus state: current_term: 1 leader_uuid: "f806e92acab647c6892e71149dd73dcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f806e92acab647c6892e71149dd73dcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38679 } } }
I20260619 16:53:31.587872 21266 sys_catalog.cc:458] T 00000000000000000000000000000000 P f806e92acab647c6892e71149dd73dcf [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:31.588074 21264 sys_catalog.cc:458] T 00000000000000000000000000000000 P f806e92acab647c6892e71149dd73dcf [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:31.588328 21273 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:53:31.588876 21273 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:53:31.590555 21273 catalog_manager.cc:1347] Generated new cluster ID: 5a8656610b744e8191c111515e0399e7
I20260619 16:53:31.590607 21273 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:53:31.594573 21273 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:53:31.594779 21273 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:53:31.600137 21273 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f806e92acab647c6892e71149dd73dcf: Generated new TSK 0
I20260619 16:53:31.600378 21273 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:53:31.606900 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 21186
2026-06-19T16:53:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1257 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:53:31 dist-test-slave-jsw9 krb5kdc[21285](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:53:31 dist-test-slave-jsw9 krb5kdc[21285](info): set up 2 sockets
Jun 19 16:53:31 dist-test-slave-jsw9 krb5kdc[21285](info): commencing operation
krb5kdc: starting...
W20260619 16:53:32.566540 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:53:32 dist-test-slave-jsw9 krb5kdc[21285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:53:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:53:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:53:32.709342 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39057
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:53:32.801623 21298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:32.801901 21298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:32.801939 21298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:32.803303 21298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:53:32.803369 21298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:32.803388 21298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:32.803406 21298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:53:32.803421 21298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:53:32.805199 21298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:32.805528 21298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:32.805795 21298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:32.811254 21303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:32.811254 21306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:32.811439 21304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:32.811497 21298 server_base.cc:1034] running on GCE node
I20260619 16:53:32.811962 21298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:32.812306 21298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:32.813496 21298 hybrid_clock.cc:648] HybridClock initialized: now 1781888012813470 us; error 39 us; skew 500 ppm
Jun 19 16:53:32 dist-test-slave-jsw9 krb5kdc[21285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:32.817472 21298 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:53:32.818040 21298 webserver.cc:460] Webserver started at http://127.0.0.1:35585/ using document root <none> and password file <none>
I20260619 16:53:32.818316 21298 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:32.818372 21298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:32.818487 21298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:32.819839 21298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "bb1e6dfc474648f9a805fbf31336b6e3"
format_stamp: "Formatted at 2026-06-19 16:53:32 on dist-test-slave-jsw9"
server_key: "1f2d9bf0fd2e13821ef0f15d7a8f9c48"
server_key_iv: "dc596a5ea3e2531ec8f9fd85a87090ff"
server_key_version: "encryptionkey@0"
I20260619 16:53:32.820191 21298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "bb1e6dfc474648f9a805fbf31336b6e3"
format_stamp: "Formatted at 2026-06-19 16:53:32 on dist-test-slave-jsw9"
server_key: "1f2d9bf0fd2e13821ef0f15d7a8f9c48"
server_key_iv: "dc596a5ea3e2531ec8f9fd85a87090ff"
server_key_version: "encryptionkey@0"
I20260619 16:53:32.821820 21298 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 16:53:32.822691 21313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:32.822889 21298 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 16:53:32.822983 21298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "bb1e6dfc474648f9a805fbf31336b6e3"
format_stamp: "Formatted at 2026-06-19 16:53:32 on dist-test-slave-jsw9"
server_key: "1f2d9bf0fd2e13821ef0f15d7a8f9c48"
server_key_iv: "dc596a5ea3e2531ec8f9fd85a87090ff"
server_key_version: "encryptionkey@0"
I20260619 16:53:32.823056 21298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:32.846006 21298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:32.848524 21298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:32.848655 21298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:32.853116 21298 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39057
I20260619 16:53:32.853199 21365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39057 every 8 connection(s)
I20260619 16:53:32.853569 21298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:53:32.854571 21366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:32.856323 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 21298
I20260619 16:53:32.856473 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:53:32.856695 392 external_mini_cluster.cc:1428] Setting key 3507b1dad70439a834dadb7750a5b662
I20260619 16:53:32.857560 21366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb1e6dfc474648f9a805fbf31336b6e3: Bootstrap starting.
I20260619 16:53:32.858378 21366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb1e6dfc474648f9a805fbf31336b6e3: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:32.858753 21366 log.cc:824] T 00000000000000000000000000000000 P bb1e6dfc474648f9a805fbf31336b6e3: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:32.859632 21366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb1e6dfc474648f9a805fbf31336b6e3: No bootstrap required, opened a new log
I20260619 16:53:32.862035 21366 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb1e6dfc474648f9a805fbf31336b6e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb1e6dfc474648f9a805fbf31336b6e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39057 } }
I20260619 16:53:32.862308 21366 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb1e6dfc474648f9a805fbf31336b6e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:32.862361 21366 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb1e6dfc474648f9a805fbf31336b6e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb1e6dfc474648f9a805fbf31336b6e3, State: Initialized, Role: FOLLOWER
I20260619 16:53:32.862567 21366 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb1e6dfc474648f9a805fbf31336b6e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb1e6dfc474648f9a805fbf31336b6e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39057 } }
I20260619 16:53:32.862653 21366 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb1e6dfc474648f9a805fbf31336b6e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:53:32.862680 21366 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb1e6dfc474648f9a805fbf31336b6e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:53:32.862732 21366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb1e6dfc474648f9a805fbf31336b6e3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:32.863441 21366 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb1e6dfc474648f9a805fbf31336b6e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb1e6dfc474648f9a805fbf31336b6e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39057 } }
I20260619 16:53:32.863575 21366 leader_election.cc:302] T 00000000000000000000000000000000 P bb1e6dfc474648f9a805fbf31336b6e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb1e6dfc474648f9a805fbf31336b6e3; no voters:
I20260619 16:53:32.863811 21366 leader_election.cc:288] T 00000000000000000000000000000000 P bb1e6dfc474648f9a805fbf31336b6e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:53:32.863893 21371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb1e6dfc474648f9a805fbf31336b6e3 [term 1 FOLLOWER]: Leader election won for term 1
Jun 19 16:53:32 dist-test-slave-jsw9 krb5kdc[21285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 16:53:32.864082 21371 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb1e6dfc474648f9a805fbf31336b6e3 [term 1 LEADER]: Becoming Leader. State: Replica: bb1e6dfc474648f9a805fbf31336b6e3, State: Running, Role: LEADER
I20260619 16:53:32.864153 21366 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb1e6dfc474648f9a805fbf31336b6e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:53:32.864174 21371 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb1e6dfc474648f9a805fbf31336b6e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb1e6dfc474648f9a805fbf31336b6e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39057 } }
I20260619 16:53:32.864636 21371 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb1e6dfc474648f9a805fbf31336b6e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb1e6dfc474648f9a805fbf31336b6e3. Latest consensus state: current_term: 1 leader_uuid: "bb1e6dfc474648f9a805fbf31336b6e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb1e6dfc474648f9a805fbf31336b6e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39057 } } }
I20260619 16:53:32.864778 21372 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb1e6dfc474648f9a805fbf31336b6e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb1e6dfc474648f9a805fbf31336b6e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb1e6dfc474648f9a805fbf31336b6e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39057 } } }
I20260619 16:53:32.864782 21371 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb1e6dfc474648f9a805fbf31336b6e3 [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:32.864852 21372 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb1e6dfc474648f9a805fbf31336b6e3 [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:32.866029 21385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:53:32.867170 21385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:53:32.869256 21385 catalog_manager.cc:1347] Generated new cluster ID: 92f7f34f483848f98a30884aade8f26f
I20260619 16:53:32.869328 21385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:53:32.870261 21369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:32.857308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.184:43801 (local address 127.0.0.1:39057)
0619 16:53:32.857503 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:32.857512 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:32.857542 (+ 30us) server_negotiation.cc:407] Connection header received
0619 16:53:32.858056 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:32.858064 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:32.858124 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:32.858310 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:32.859056 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:32.860076 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:32.860874 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:32.861468 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:32.861607 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:32.864833 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:32.864840 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:32.864841 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:32.864858 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:32.867464 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:32.868524 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:32.868529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:32.868529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:32.868672 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:32.868996 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:32.868997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:32.868998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:32.869364 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:32.869531 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:32.869849 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 16:53:32.869983 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":96,"threads_started":1}
I20260619 16:53:32.871414 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:53:32.878163 21385 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:53:32.878934 21385 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:53:32.883488 21369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:32.873594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.184:37553 (local address 127.0.0.1:39057)
0619 16:53:32.873638 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:32.873640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:32.873650 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:53:32.873722 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:32.873723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:32.873735 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:32.873792 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:32.874630 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:32.875426 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:32.876534 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:32.876898 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:32.876969 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:32.878673 (+ 1704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:32.878675 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:32.878676 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:32.878685 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:32.880380 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:32.880968 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:32.880969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:32.880969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:32.881031 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:32.881247 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:32.881248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:32.881248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:32.881387 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:32.881492 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:32.881624 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 16:53:32.881689 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 16:53:32.892246 21385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb1e6dfc474648f9a805fbf31336b6e3: Generated new TSK 0
I20260619 16:53:32.892606 21385 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:53:32.895677 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 21298
W20260619 16:53:32.900588 21283 connection.cc:535] client connection to 127.0.0.1:39057 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39057): Connection reset by peer (error 104)
2026-06-19T16:53:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1289 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:53:32 dist-test-slave-jsw9 krb5kdc[21397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:53:32 dist-test-slave-jsw9 krb5kdc[21397](info): set up 2 sockets
Jun 19 16:53:32 dist-test-slave-jsw9 krb5kdc[21397](info): commencing operation
krb5kdc: starting...
W20260619 16:53:33.878548 392 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:53:33 dist-test-slave-jsw9 krb5kdc[21397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:53:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:53:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:53:34.014524 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38355
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:53:34.106385 21410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:34.106678 21410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:34.106716 21410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:34.108129 21410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:53:34.108201 21410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:34.108222 21410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:34.108238 21410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:53:34.108253 21410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:53:34.110008 21410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:34.110359 21410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:34.110622 21410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:34.116128 21416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:34.116139 21415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:34.116112 21418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:34.116269 21410 server_base.cc:1034] running on GCE node
I20260619 16:53:34.116652 21410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:34.117050 21410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:34.118232 21410 hybrid_clock.cc:648] HybridClock initialized: now 1781888014118214 us; error 33 us; skew 500 ppm
Jun 19 16:53:34 dist-test-slave-jsw9 krb5kdc[21397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:34.120966 21410 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:53:34.121510 21410 webserver.cc:460] Webserver started at http://127.0.0.1:45449/ using document root <none> and password file <none>
I20260619 16:53:34.121742 21410 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:34.121793 21410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:34.121930 21410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:34.123190 21410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "3447c1a1ccd842398eb31c1bedbeddc0"
format_stamp: "Formatted at 2026-06-19 16:53:34 on dist-test-slave-jsw9"
server_key: "7bc4959648240450d43bab4280ed60e5"
server_key_iv: "c67e4fe13ab4008f4d4c6fc4838fbd6e"
server_key_version: "encryptionkey@0"
I20260619 16:53:34.123555 21410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "3447c1a1ccd842398eb31c1bedbeddc0"
format_stamp: "Formatted at 2026-06-19 16:53:34 on dist-test-slave-jsw9"
server_key: "7bc4959648240450d43bab4280ed60e5"
server_key_iv: "c67e4fe13ab4008f4d4c6fc4838fbd6e"
server_key_version: "encryptionkey@0"
I20260619 16:53:34.124966 21410 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 16:53:34.125820 21425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:34.126029 21410 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:53:34.126121 21410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "3447c1a1ccd842398eb31c1bedbeddc0"
format_stamp: "Formatted at 2026-06-19 16:53:34 on dist-test-slave-jsw9"
server_key: "7bc4959648240450d43bab4280ed60e5"
server_key_iv: "c67e4fe13ab4008f4d4c6fc4838fbd6e"
server_key_version: "encryptionkey@0"
I20260619 16:53:34.126226 21410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:34.140455 21410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:34.142962 21410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:34.143136 21410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:34.147902 21410 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38355
I20260619 16:53:34.147945 21477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38355 every 8 connection(s)
I20260619 16:53:34.148342 21410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:53:34.149376 21478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:34.151010 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 21410
I20260619 16:53:34.151139 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:53:34.151422 392 external_mini_cluster.cc:1428] Setting key 51eebfbc620e2e7afe118168aac74acf
I20260619 16:53:34.152499 21478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3447c1a1ccd842398eb31c1bedbeddc0: Bootstrap starting.
I20260619 16:53:34.153163 21478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3447c1a1ccd842398eb31c1bedbeddc0: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:34.153529 21478 log.cc:824] T 00000000000000000000000000000000 P 3447c1a1ccd842398eb31c1bedbeddc0: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:34.154457 21478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3447c1a1ccd842398eb31c1bedbeddc0: No bootstrap required, opened a new log
I20260619 16:53:34.156992 21478 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3447c1a1ccd842398eb31c1bedbeddc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3447c1a1ccd842398eb31c1bedbeddc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38355 } }
I20260619 16:53:34.157227 21478 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3447c1a1ccd842398eb31c1bedbeddc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:34.157271 21478 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3447c1a1ccd842398eb31c1bedbeddc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3447c1a1ccd842398eb31c1bedbeddc0, State: Initialized, Role: FOLLOWER
I20260619 16:53:34.157476 21478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3447c1a1ccd842398eb31c1bedbeddc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3447c1a1ccd842398eb31c1bedbeddc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38355 } }
I20260619 16:53:34.157570 21478 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3447c1a1ccd842398eb31c1bedbeddc0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:53:34.157608 21478 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3447c1a1ccd842398eb31c1bedbeddc0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:53:34.157672 21478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3447c1a1ccd842398eb31c1bedbeddc0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:34.158885 21478 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3447c1a1ccd842398eb31c1bedbeddc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3447c1a1ccd842398eb31c1bedbeddc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38355 } }
I20260619 16:53:34.158967 21481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:34.152068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.184:43289 (local address 127.0.0.1:38355)
0619 16:53:34.152288 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:34.152300 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:34.152339 (+ 39us) server_negotiation.cc:407] Connection header received
0619 16:53:34.152984 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:34.152995 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:34.153071 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:34.153234 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:34.153675 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:34.155197 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:34.156468 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:34.157177 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:34.157318 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:34.158177 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:34.158182 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:34.158209 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:34.158449 (+ 240us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:34.158494 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:34.158511 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:34.158602 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:34.158677 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 16:53:34.158712 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":86,"threads_started":1}
I20260619 16:53:34.159026 21478 leader_election.cc:302] T 00000000000000000000000000000000 P 3447c1a1ccd842398eb31c1bedbeddc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3447c1a1ccd842398eb31c1bedbeddc0; no voters:
I20260619 16:53:34.159194 21478 leader_election.cc:288] T 00000000000000000000000000000000 P 3447c1a1ccd842398eb31c1bedbeddc0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:53:34.159265 21483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3447c1a1ccd842398eb31c1bedbeddc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:53:34.159466 21483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3447c1a1ccd842398eb31c1bedbeddc0 [term 1 LEADER]: Becoming Leader. State: Replica: 3447c1a1ccd842398eb31c1bedbeddc0, State: Running, Role: LEADER
I20260619 16:53:34.159479 21478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3447c1a1ccd842398eb31c1bedbeddc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:53:34.159709 21483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3447c1a1ccd842398eb31c1bedbeddc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3447c1a1ccd842398eb31c1bedbeddc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38355 } }
I20260619 16:53:34.160373 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:53:34.160406 21485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3447c1a1ccd842398eb31c1bedbeddc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3447c1a1ccd842398eb31c1bedbeddc0. Latest consensus state: current_term: 1 leader_uuid: "3447c1a1ccd842398eb31c1bedbeddc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3447c1a1ccd842398eb31c1bedbeddc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38355 } } }
I20260619 16:53:34.160492 21485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3447c1a1ccd842398eb31c1bedbeddc0 [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:34.160688 21484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3447c1a1ccd842398eb31c1bedbeddc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3447c1a1ccd842398eb31c1bedbeddc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3447c1a1ccd842398eb31c1bedbeddc0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38355 } } }
I20260619 16:53:34.160742 21484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3447c1a1ccd842398eb31c1bedbeddc0 [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:34.161062 21490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:53:34.161619 21490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:53:34.163532 21490 catalog_manager.cc:1347] Generated new cluster ID: 149675cf9099438d94bca56f270252e2
I20260619 16:53:34.163604 21490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:53:34.174991 21481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:34.162388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.184:53903 (local address 127.0.0.1:38355)
0619 16:53:34.162432 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:34.162434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:34.170520 (+ 8086us) server_negotiation.cc:407] Connection header received
0619 16:53:34.170624 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:34.170626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:34.170644 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:34.170701 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:34.170896 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:34.171720 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:34.172916 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:34.173270 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:34.173335 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:34.174675 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:34.174677 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:34.174678 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:34.174685 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:34.174706 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:34.174712 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:34.174782 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:34.174871 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 16:53:34.174885 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 16:53:34.177639 21490 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:53:34.178467 21490 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:53:34.184777 21490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3447c1a1ccd842398eb31c1bedbeddc0: Generated new TSK 0
I20260619 16:53:34.185024 21490 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:53:34.189157 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 21410
W20260619 16:53:34.194015 21395 connection.cc:535] client connection to 127.0.0.1:38355 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38355): Connection reset by peer (error 104)
2026-06-19T16:53:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1293 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:53:34 dist-test-slave-jsw9 krb5kdc[21509](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:53:34 dist-test-slave-jsw9 krb5kdc[21509](info): set up 2 sockets
Jun 19 16:53:34 dist-test-slave-jsw9 krb5kdc[21509](info): commencing operation
krb5kdc: starting...
W20260619 16:53:35.178534 392 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:53:35 dist-test-slave-jsw9 krb5kdc[21509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:53:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:53:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:53:35.321727 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46777
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:53:35.415082 21522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:35.415380 21522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:35.415417 21522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:35.416788 21522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:53:35.416848 21522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:35.416867 21522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:35.416886 21522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:53:35.416901 21522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:53:35.418872 21522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:35.419234 21522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:35.419508 21522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:35.425364 21530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:35.425485 21527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:35.425539 21528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:35.425580 21522 server_base.cc:1034] running on GCE node
I20260619 16:53:35.426025 21522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:35.426436 21522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:35.427621 21522 hybrid_clock.cc:648] HybridClock initialized: now 1781888015427518 us; error 125 us; skew 500 ppm
Jun 19 16:53:35 dist-test-slave-jsw9 krb5kdc[21509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888015, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:35.435974 21522 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:53:35.436619 21522 webserver.cc:460] Webserver started at http://127.0.0.1:35629/ using document root <none> and password file <none>
I20260619 16:53:35.436877 21522 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:35.436935 21522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:35.437220 21522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:35.438589 21522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "f19791417ac749529bad4da4c4a30fc9"
format_stamp: "Formatted at 2026-06-19 16:53:35 on dist-test-slave-jsw9"
server_key: "0a97d9b12a2f7f30b017b7fa787689b9"
server_key_iv: "0d23da80f40b73988a3ae8c8fe05829e"
server_key_version: "encryptionkey@0"
I20260619 16:53:35.438983 21522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "f19791417ac749529bad4da4c4a30fc9"
format_stamp: "Formatted at 2026-06-19 16:53:35 on dist-test-slave-jsw9"
server_key: "0a97d9b12a2f7f30b017b7fa787689b9"
server_key_iv: "0d23da80f40b73988a3ae8c8fe05829e"
server_key_version: "encryptionkey@0"
I20260619 16:53:35.440496 21522 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 16:53:35.441417 21537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:35.441707 21522 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:53:35.441838 21522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "f19791417ac749529bad4da4c4a30fc9"
format_stamp: "Formatted at 2026-06-19 16:53:35 on dist-test-slave-jsw9"
server_key: "0a97d9b12a2f7f30b017b7fa787689b9"
server_key_iv: "0d23da80f40b73988a3ae8c8fe05829e"
server_key_version: "encryptionkey@0"
I20260619 16:53:35.441926 21522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:35.445874 21522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:35.448148 21522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:35.448315 21522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:35.452984 21522 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46777
I20260619 16:53:35.453060 21589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46777 every 8 connection(s)
I20260619 16:53:35.453413 21522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:53:35.454624 21590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:35.457365 21590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f19791417ac749529bad4da4c4a30fc9: Bootstrap starting.
I20260619 16:53:35.457935 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 21522
I20260619 16:53:35.458037 21590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f19791417ac749529bad4da4c4a30fc9: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:35.458074 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:53:35.458348 392 external_mini_cluster.cc:1428] Setting key 20bdf39b0005551a9a3d9dd0525ca393
I20260619 16:53:35.458375 21590 log.cc:824] T 00000000000000000000000000000000 P f19791417ac749529bad4da4c4a30fc9: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:35.459246 21590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f19791417ac749529bad4da4c4a30fc9: No bootstrap required, opened a new log
I20260619 16:53:35.461310 21594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:35.458962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.184:36615 (local address 127.0.0.1:46777)
0619 16:53:35.459180 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:35.459188 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:35.459216 (+ 28us) server_negotiation.cc:407] Connection header received
0619 16:53:35.459732 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:35.459740 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:35.459770 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:35.459920 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:35.460704 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:35.460708 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:35.460708 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:35.460712 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:35.460755 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:35.460767 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:35.460874 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:35.460907 (+ 33us) server_negotiation.cc:299] Negotiation successful
0619 16:53:35.460957 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":101,"threads_started":1}
I20260619 16:53:35.461330 21590 raft_consensus.cc:348] T 00000000000000000000000000000000 P f19791417ac749529bad4da4c4a30fc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f19791417ac749529bad4da4c4a30fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46777 } }
I20260619 16:53:35.461604 21590 raft_consensus.cc:374] T 00000000000000000000000000000000 P f19791417ac749529bad4da4c4a30fc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:35.461651 21590 raft_consensus.cc:729] T 00000000000000000000000000000000 P f19791417ac749529bad4da4c4a30fc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19791417ac749529bad4da4c4a30fc9, State: Initialized, Role: FOLLOWER
I20260619 16:53:35.461796 21590 consensus_queue.cc:260] T 00000000000000000000000000000000 P f19791417ac749529bad4da4c4a30fc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f19791417ac749529bad4da4c4a30fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46777 } }
I20260619 16:53:35.461910 21590 raft_consensus.cc:388] T 00000000000000000000000000000000 P f19791417ac749529bad4da4c4a30fc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:53:35.461988 21590 raft_consensus.cc:482] T 00000000000000000000000000000000 P f19791417ac749529bad4da4c4a30fc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:53:35.462046 21590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f19791417ac749529bad4da4c4a30fc9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:35.462378 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:53:35.462814 21590 raft_consensus.cc:504] T 00000000000000000000000000000000 P f19791417ac749529bad4da4c4a30fc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f19791417ac749529bad4da4c4a30fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46777 } }
I20260619 16:53:35.462939 21590 leader_election.cc:302] T 00000000000000000000000000000000 P f19791417ac749529bad4da4c4a30fc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f19791417ac749529bad4da4c4a30fc9; no voters:
I20260619 16:53:35.463128 21590 leader_election.cc:288] T 00000000000000000000000000000000 P f19791417ac749529bad4da4c4a30fc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:53:35.463189 21595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f19791417ac749529bad4da4c4a30fc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:53:35.463392 21595 raft_consensus.cc:686] T 00000000000000000000000000000000 P f19791417ac749529bad4da4c4a30fc9 [term 1 LEADER]: Becoming Leader. State: Replica: f19791417ac749529bad4da4c4a30fc9, State: Running, Role: LEADER
I20260619 16:53:35.463416 21590 sys_catalog.cc:564] T 00000000000000000000000000000000 P f19791417ac749529bad4da4c4a30fc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:53:35.463560 21595 consensus_queue.cc:237] T 00000000000000000000000000000000 P f19791417ac749529bad4da4c4a30fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f19791417ac749529bad4da4c4a30fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46777 } }
I20260619 16:53:35.463986 21596 sys_catalog.cc:455] T 00000000000000000000000000000000 P f19791417ac749529bad4da4c4a30fc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f19791417ac749529bad4da4c4a30fc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f19791417ac749529bad4da4c4a30fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46777 } } }
I20260619 16:53:35.464069 21596 sys_catalog.cc:458] T 00000000000000000000000000000000 P f19791417ac749529bad4da4c4a30fc9 [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:35.464305 21597 sys_catalog.cc:455] T 00000000000000000000000000000000 P f19791417ac749529bad4da4c4a30fc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f19791417ac749529bad4da4c4a30fc9. Latest consensus state: current_term: 1 leader_uuid: "f19791417ac749529bad4da4c4a30fc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f19791417ac749529bad4da4c4a30fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46777 } } }
I20260619 16:53:35.464426 21597 sys_catalog.cc:458] T 00000000000000000000000000000000 P f19791417ac749529bad4da4c4a30fc9 [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:35.465226 21607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:53:35.465381 21594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:35.464012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.184:51231 (local address 127.0.0.1:46777)
0619 16:53:35.464054 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:35.464057 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:35.464279 (+ 222us) server_negotiation.cc:407] Connection header received
0619 16:53:35.464388 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:35.464389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:35.464395 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:35.464422 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:35.465134 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:35.465136 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:35.465136 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:35.465140 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:35.465157 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:35.465162 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:35.465204 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:35.465304 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 16:53:35.465317 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 16:53:35.466074 21607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:53:35.467947 21607 catalog_manager.cc:1347] Generated new cluster ID: 6fa0cce63d034c9d992df0870c8911b9
I20260619 16:53:35.468007 21607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:53:35.485934 21607 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:53:35.486159 21607 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:53:35.488934 21607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f19791417ac749529bad4da4c4a30fc9: Generated new TSK 0
I20260619 16:53:35.489137 21607 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:53:35.493975 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 21522
2026-06-19T16:53:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1304 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:53:35 dist-test-slave-jsw9 krb5kdc[21621](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:53:35 dist-test-slave-jsw9 krb5kdc[21621](info): set up 2 sockets
Jun 19 16:53:35 dist-test-slave-jsw9 krb5kdc[21621](info): commencing operation
krb5kdc: starting...
W20260619 16:53:36.494722 392 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:53:36 dist-test-slave-jsw9 krb5kdc[21621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:53:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:53:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:53:36.629315 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38677
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:53:36.722904 21634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:36.723183 21634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:36.723214 21634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:36.724485 21634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:53:36.724535 21634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:36.724548 21634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:36.724560 21634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:53:36.724570 21634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:53:36.726256 21634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38677
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:36.726511 21634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:36.726728 21634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:36.731871 21639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:36.731822 21640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:36.731859 21634 server_base.cc:1034] running on GCE node
W20260619 16:53:36.731841 21642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:36.732331 21634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:36.732666 21634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:36.733835 21634 hybrid_clock.cc:648] HybridClock initialized: now 1781888016733815 us; error 42 us; skew 500 ppm
Jun 19 16:53:36 dist-test-slave-jsw9 krb5kdc[21621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888016, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:36.736712 21634 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:53:36.737226 21634 webserver.cc:460] Webserver started at http://127.0.0.1:39013/ using document root <none> and password file <none>
I20260619 16:53:36.737448 21634 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:36.737496 21634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:36.737623 21634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:36.739053 21634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "92873978b42c4df3b5d6eb4ec44a1aee"
format_stamp: "Formatted at 2026-06-19 16:53:36 on dist-test-slave-jsw9"
server_key: "b515cec1b87b84bc7165a61778dc8982"
server_key_iv: "97408157aae56938732f88df76043343"
server_key_version: "encryptionkey@0"
I20260619 16:53:36.739588 21634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "92873978b42c4df3b5d6eb4ec44a1aee"
format_stamp: "Formatted at 2026-06-19 16:53:36 on dist-test-slave-jsw9"
server_key: "b515cec1b87b84bc7165a61778dc8982"
server_key_iv: "97408157aae56938732f88df76043343"
server_key_version: "encryptionkey@0"
I20260619 16:53:36.741477 21634 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:53:36.742741 21649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:36.742937 21634 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:53:36.743048 21634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "92873978b42c4df3b5d6eb4ec44a1aee"
format_stamp: "Formatted at 2026-06-19 16:53:36 on dist-test-slave-jsw9"
server_key: "b515cec1b87b84bc7165a61778dc8982"
server_key_iv: "97408157aae56938732f88df76043343"
server_key_version: "encryptionkey@0"
I20260619 16:53:36.743136 21634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:36.756162 21634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:36.758509 21634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:36.758659 21634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:36.763468 21634 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38677
I20260619 16:53:36.763504 21701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38677 every 8 connection(s)
I20260619 16:53:36.763926 21634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:53:36.764892 21702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:36.765508 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 21634
I20260619 16:53:36.765625 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:53:36.765834 392 external_mini_cluster.cc:1428] Setting key 9f3fe4eb9251ae965b4f8c3d52f6a3a8
I20260619 16:53:36.767899 21702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92873978b42c4df3b5d6eb4ec44a1aee: Bootstrap starting.
I20260619 16:53:36.768556 21702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92873978b42c4df3b5d6eb4ec44a1aee: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:36.768853 21702 log.cc:824] T 00000000000000000000000000000000 P 92873978b42c4df3b5d6eb4ec44a1aee: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:36.769543 21702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92873978b42c4df3b5d6eb4ec44a1aee: No bootstrap required, opened a new log
I20260619 16:53:36.771705 21702 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92873978b42c4df3b5d6eb4ec44a1aee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92873978b42c4df3b5d6eb4ec44a1aee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38677 } }
I20260619 16:53:36.771881 21702 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92873978b42c4df3b5d6eb4ec44a1aee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:36.771931 21702 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92873978b42c4df3b5d6eb4ec44a1aee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92873978b42c4df3b5d6eb4ec44a1aee, State: Initialized, Role: FOLLOWER
I20260619 16:53:36.772082 21702 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92873978b42c4df3b5d6eb4ec44a1aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92873978b42c4df3b5d6eb4ec44a1aee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38677 } }
I20260619 16:53:36.772161 21702 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92873978b42c4df3b5d6eb4ec44a1aee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:53:36.772204 21702 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92873978b42c4df3b5d6eb4ec44a1aee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:53:36.772254 21702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92873978b42c4df3b5d6eb4ec44a1aee [term 0 FOLLOWER]: Advancing to term 1
Jun 19 16:53:36 dist-test-slave-jsw9 krb5kdc[21621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 16:53:36.773095 21702 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92873978b42c4df3b5d6eb4ec44a1aee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92873978b42c4df3b5d6eb4ec44a1aee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38677 } }
I20260619 16:53:36.773205 21702 leader_election.cc:302] T 00000000000000000000000000000000 P 92873978b42c4df3b5d6eb4ec44a1aee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92873978b42c4df3b5d6eb4ec44a1aee; no voters:
I20260619 16:53:36.773416 21702 leader_election.cc:288] T 00000000000000000000000000000000 P 92873978b42c4df3b5d6eb4ec44a1aee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:53:36.773468 21707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92873978b42c4df3b5d6eb4ec44a1aee [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:53:36.773643 21707 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92873978b42c4df3b5d6eb4ec44a1aee [term 1 LEADER]: Becoming Leader. State: Replica: 92873978b42c4df3b5d6eb4ec44a1aee, State: Running, Role: LEADER
I20260619 16:53:36.773731 21702 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92873978b42c4df3b5d6eb4ec44a1aee [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:53:36.773779 21707 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92873978b42c4df3b5d6eb4ec44a1aee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92873978b42c4df3b5d6eb4ec44a1aee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38677 } }
I20260619 16:53:36.774171 21707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92873978b42c4df3b5d6eb4ec44a1aee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92873978b42c4df3b5d6eb4ec44a1aee. Latest consensus state: current_term: 1 leader_uuid: "92873978b42c4df3b5d6eb4ec44a1aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92873978b42c4df3b5d6eb4ec44a1aee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38677 } } }
I20260619 16:53:36.774356 21707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92873978b42c4df3b5d6eb4ec44a1aee [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:36.774405 21708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92873978b42c4df3b5d6eb4ec44a1aee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92873978b42c4df3b5d6eb4ec44a1aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92873978b42c4df3b5d6eb4ec44a1aee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38677 } } }
I20260619 16:53:36.774549 21708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92873978b42c4df3b5d6eb4ec44a1aee [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:36.775380 21716 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:53:36.776010 21716 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:53:36.777668 21716 catalog_manager.cc:1347] Generated new cluster ID: df956a90a07c4144af20d23f8f7f4fb4
I20260619 16:53:36.777715 21716 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:53:36.777908 21705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:36.766435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.184:55165 (local address 127.0.0.1:38677)
0619 16:53:36.766648 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:36.766658 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:36.766685 (+ 27us) server_negotiation.cc:407] Connection header received
0619 16:53:36.767128 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:36.767134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:36.767187 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:36.767376 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:36.768067 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:36.769070 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:36.770020 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:36.770552 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:36.770667 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:36.773697 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:36.773703 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:36.773704 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:36.773719 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:36.775889 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:36.776403 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:36.776407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:36.776407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:36.776519 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:36.776779 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:36.776780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:36.776780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:36.777114 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:36.777283 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:36.777593 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 16:53:36.777701 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":81,"threads_started":1}
I20260619 16:53:36.778779 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:53:36.794996 21705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:36.780548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.184:43561 (local address 127.0.0.1:38677)
0619 16:53:36.780585 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:36.780587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:36.786489 (+ 5902us) server_negotiation.cc:407] Connection header received
0619 16:53:36.786565 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:36.786566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:36.786580 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:36.786657 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:36.787438 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:36.788265 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:36.789436 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:36.789785 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:36.789846 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:36.791508 (+ 1662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:36.791509 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:36.791510 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:36.791518 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:36.793566 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:36.794054 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:36.794055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:36.794056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:36.794118 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:36.794417 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:36.794418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:36.794419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:36.794561 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:36.794667 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:36.794816 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 16:53:36.794885 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 16:53:36.805378 21716 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:53:36.805995 21716 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:53:36.810361 21716 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92873978b42c4df3b5d6eb4ec44a1aee: Generated new TSK 0
I20260619 16:53:36.810576 21716 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:53:36.818817 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 21634
W20260619 16:53:36.823553 21619 connection.cc:535] client connection to 127.0.0.1:38677 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38677): Connection reset by peer (error 104)
2026-06-19T16:53:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:53:36 dist-test-slave-jsw9 krb5kdc[21733](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:53:36 dist-test-slave-jsw9 krb5kdc[21733](info): set up 2 sockets
Jun 19 16:53:36 dist-test-slave-jsw9 krb5kdc[21733](info): commencing operation
krb5kdc: starting...
W20260619 16:53:37.799045 392 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:53:37 dist-test-slave-jsw9 krb5kdc[21733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:53:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:53:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:53:37.956442 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45663
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:53:38.056923 21746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:38.057441 21746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:38.057476 21746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:38.059628 21746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:53:38.059724 21746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:38.059741 21746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:38.059753 21746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:53:38.059765 21746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:53:38.061767 21746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45663
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:38.062120 21746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:38.062445 21746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:53:38.068598 21746 server_base.cc:1034] running on GCE node
W20260619 16:53:38.068619 21751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:38.068559 21754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:38.068591 21752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:38.069031 21746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:38.069415 21746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:38.070595 21746 hybrid_clock.cc:648] HybridClock initialized: now 1781888018070559 us; error 56 us; skew 500 ppm
Jun 19 16:53:38 dist-test-slave-jsw9 krb5kdc[21733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:38.073858 21746 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:53:38.074463 21746 webserver.cc:460] Webserver started at http://127.0.0.1:41083/ using document root <none> and password file <none>
I20260619 16:53:38.074739 21746 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:38.074786 21746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:38.074893 21746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:38.076121 21746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "be2d0b5bb64b4a5d8135c2654eec2d84"
format_stamp: "Formatted at 2026-06-19 16:53:38 on dist-test-slave-jsw9"
server_key: "43d6a4820d11ee6ba148e9e5566677aa"
server_key_iv: "983c2b57e491b5b93bc100961f8c21d4"
server_key_version: "encryptionkey@0"
I20260619 16:53:38.076480 21746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "be2d0b5bb64b4a5d8135c2654eec2d84"
format_stamp: "Formatted at 2026-06-19 16:53:38 on dist-test-slave-jsw9"
server_key: "43d6a4820d11ee6ba148e9e5566677aa"
server_key_iv: "983c2b57e491b5b93bc100961f8c21d4"
server_key_version: "encryptionkey@0"
I20260619 16:53:38.078029 21746 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 16:53:38.079221 21761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:38.079594 21746 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:53:38.079728 21746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "be2d0b5bb64b4a5d8135c2654eec2d84"
format_stamp: "Formatted at 2026-06-19 16:53:38 on dist-test-slave-jsw9"
server_key: "43d6a4820d11ee6ba148e9e5566677aa"
server_key_iv: "983c2b57e491b5b93bc100961f8c21d4"
server_key_version: "encryptionkey@0"
I20260619 16:53:38.079929 21746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:38.102667 21746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:38.105710 21746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:38.105919 21746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:38.111263 21746 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45663
I20260619 16:53:38.111361 21813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45663 every 8 connection(s)
I20260619 16:53:38.111748 21746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:53:38.113099 21814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:38.113590 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 21746
I20260619 16:53:38.113720 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:53:38.113941 392 external_mini_cluster.cc:1428] Setting key 69fc8ea8273bc4418b62c3cf7c4c5d80
I20260619 16:53:38.116611 21814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be2d0b5bb64b4a5d8135c2654eec2d84: Bootstrap starting.
I20260619 16:53:38.117412 21814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be2d0b5bb64b4a5d8135c2654eec2d84: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:38.117897 21814 log.cc:824] T 00000000000000000000000000000000 P be2d0b5bb64b4a5d8135c2654eec2d84: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:38.119782 21814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be2d0b5bb64b4a5d8135c2654eec2d84: No bootstrap required, opened a new log
I20260619 16:53:38.122084 21817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:38.114812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.184:44657 (local address 127.0.0.1:45663)
0619 16:53:38.115006 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:38.115017 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:38.115049 (+ 32us) server_negotiation.cc:407] Connection header received
0619 16:53:38.115582 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:38.115591 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:38.115652 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:38.115797 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:38.116098 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:38.117457 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:38.118923 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:38.120251 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:38.120389 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:38.121197 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:38.121203 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:38.121203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:38.121469 (+ 266us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:38.121517 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:38.121536 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:38.121639 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:38.121762 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 16:53:38.121798 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":76,"threads_started":1}
I20260619 16:53:38.122618 21814 raft_consensus.cc:348] T 00000000000000000000000000000000 P be2d0b5bb64b4a5d8135c2654eec2d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be2d0b5bb64b4a5d8135c2654eec2d84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45663 } }
I20260619 16:53:38.122789 21814 raft_consensus.cc:374] T 00000000000000000000000000000000 P be2d0b5bb64b4a5d8135c2654eec2d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:38.122812 21814 raft_consensus.cc:729] T 00000000000000000000000000000000 P be2d0b5bb64b4a5d8135c2654eec2d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be2d0b5bb64b4a5d8135c2654eec2d84, State: Initialized, Role: FOLLOWER
I20260619 16:53:38.122938 21814 consensus_queue.cc:260] T 00000000000000000000000000000000 P be2d0b5bb64b4a5d8135c2654eec2d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be2d0b5bb64b4a5d8135c2654eec2d84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45663 } }
I20260619 16:53:38.123019 21814 raft_consensus.cc:388] T 00000000000000000000000000000000 P be2d0b5bb64b4a5d8135c2654eec2d84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:53:38.123049 21814 raft_consensus.cc:482] T 00000000000000000000000000000000 P be2d0b5bb64b4a5d8135c2654eec2d84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:53:38.123102 21814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be2d0b5bb64b4a5d8135c2654eec2d84 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:38.123406 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:53:38.123925 21814 raft_consensus.cc:504] T 00000000000000000000000000000000 P be2d0b5bb64b4a5d8135c2654eec2d84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be2d0b5bb64b4a5d8135c2654eec2d84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45663 } }
I20260619 16:53:38.124034 21814 leader_election.cc:302] T 00000000000000000000000000000000 P be2d0b5bb64b4a5d8135c2654eec2d84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be2d0b5bb64b4a5d8135c2654eec2d84; no voters:
I20260619 16:53:38.124296 21814 leader_election.cc:288] T 00000000000000000000000000000000 P be2d0b5bb64b4a5d8135c2654eec2d84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:53:38.124361 21819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be2d0b5bb64b4a5d8135c2654eec2d84 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:53:38.124577 21819 raft_consensus.cc:686] T 00000000000000000000000000000000 P be2d0b5bb64b4a5d8135c2654eec2d84 [term 1 LEADER]: Becoming Leader. State: Replica: be2d0b5bb64b4a5d8135c2654eec2d84, State: Running, Role: LEADER
I20260619 16:53:38.124594 21814 sys_catalog.cc:564] T 00000000000000000000000000000000 P be2d0b5bb64b4a5d8135c2654eec2d84 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:53:38.124760 21819 consensus_queue.cc:237] T 00000000000000000000000000000000 P be2d0b5bb64b4a5d8135c2654eec2d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be2d0b5bb64b4a5d8135c2654eec2d84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45663 } }
I20260619 16:53:38.125190 21820 sys_catalog.cc:455] T 00000000000000000000000000000000 P be2d0b5bb64b4a5d8135c2654eec2d84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be2d0b5bb64b4a5d8135c2654eec2d84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be2d0b5bb64b4a5d8135c2654eec2d84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45663 } } }
I20260619 16:53:38.125274 21820 sys_catalog.cc:458] T 00000000000000000000000000000000 P be2d0b5bb64b4a5d8135c2654eec2d84 [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:38.125458 21822 sys_catalog.cc:455] T 00000000000000000000000000000000 P be2d0b5bb64b4a5d8135c2654eec2d84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be2d0b5bb64b4a5d8135c2654eec2d84. Latest consensus state: current_term: 1 leader_uuid: "be2d0b5bb64b4a5d8135c2654eec2d84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be2d0b5bb64b4a5d8135c2654eec2d84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45663 } } }
I20260619 16:53:38.125577 21822 sys_catalog.cc:458] T 00000000000000000000000000000000 P be2d0b5bb64b4a5d8135c2654eec2d84 [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:38.125734 21831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:53:38.126353 21831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:53:38.128574 21831 catalog_manager.cc:1347] Generated new cluster ID: 9f37d20ed8694fee913b4e4b60c1338e
I20260619 16:53:38.128646 21831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:53:38.130002 21817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:38.125178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.184:55015 (local address 127.0.0.1:45663)
0619 16:53:38.125226 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:38.125229 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:38.126156 (+ 927us) server_negotiation.cc:407] Connection header received
0619 16:53:38.126328 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:38.126329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:38.126344 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:38.126412 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:38.126651 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:38.127376 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:38.128434 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:38.128771 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:38.128858 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:38.129697 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:38.129699 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:38.129699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:38.129707 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:38.129727 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:38.129732 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:38.129806 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:38.129869 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 16:53:38.129883 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 16:53:38.144336 21831 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:53:38.144963 21831 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:53:38.156708 21831 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be2d0b5bb64b4a5d8135c2654eec2d84: Generated new TSK 0
I20260619 16:53:38.157428 21831 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:53:38.161446 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 21746
W20260619 16:53:38.166256 21731 connection.cc:535] client connection to 127.0.0.1:45663 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45663): Connection reset by peer (error 104)
2026-06-19T16:53:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1344 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:53:38 dist-test-slave-jsw9 krb5kdc[21845](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:53:38 dist-test-slave-jsw9 krb5kdc[21845](info): set up 2 sockets
Jun 19 16:53:38 dist-test-slave-jsw9 krb5kdc[21845](info): commencing operation
krb5kdc: starting...
W20260619 16:53:39.162560 392 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:53:39 dist-test-slave-jsw9 krb5kdc[21845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:53:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:53:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:53:39.298364 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40593
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:53:39.393404 21858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:39.393695 21858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:39.393724 21858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:39.395236 21858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:53:39.395316 21858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:39.395332 21858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:39.395344 21858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:53:39.395355 21858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:53:39.397210 21858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40593
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887741903642-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:39.397521 21858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:39.397789 21858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:53:39.403414 21858 server_base.cc:1034] running on GCE node
W20260619 16:53:39.403298 21866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:39.403352 21864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:39.403298 21863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:39.403870 21858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:39.404225 21858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:39.405373 21858 hybrid_clock.cc:648] HybridClock initialized: now 1781888019405357 us; error 31 us; skew 500 ppm
Jun 19 16:53:39 dist-test-slave-jsw9 krb5kdc[21845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888019, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:39.408293 21858 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:53:39.408833 21858 webserver.cc:460] Webserver started at http://127.0.0.1:37901/ using document root <none> and password file <none>
I20260619 16:53:39.409065 21858 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:39.409117 21858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:39.409265 21858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:39.410605 21858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "54d396362df54828a25574911fab8eb8"
format_stamp: "Formatted at 2026-06-19 16:53:39 on dist-test-slave-jsw9"
server_key: "511f55f510dc1f7f41193e5df768a340"
server_key_iv: "a70301a8cfa94bae1a3846db83023b53"
server_key_version: "encryptionkey@0"
I20260619 16:53:39.410984 21858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "54d396362df54828a25574911fab8eb8"
format_stamp: "Formatted at 2026-06-19 16:53:39 on dist-test-slave-jsw9"
server_key: "511f55f510dc1f7f41193e5df768a340"
server_key_iv: "a70301a8cfa94bae1a3846db83023b53"
server_key_version: "encryptionkey@0"
I20260619 16:53:39.412499 21858 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 16:53:39.413327 21873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:39.413507 21858 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:53:39.413599 21858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "54d396362df54828a25574911fab8eb8"
format_stamp: "Formatted at 2026-06-19 16:53:39 on dist-test-slave-jsw9"
server_key: "511f55f510dc1f7f41193e5df768a340"
server_key_iv: "a70301a8cfa94bae1a3846db83023b53"
server_key_version: "encryptionkey@0"
I20260619 16:53:39.413666 21858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:39.417393 21858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:39.420038 21858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:39.420208 21858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:39.425129 21858 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40593
I20260619 16:53:39.425163 21925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40593 every 8 connection(s)
I20260619 16:53:39.425606 21858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:53:39.426874 21926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:39.429736 21926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54d396362df54828a25574911fab8eb8: Bootstrap starting.
I20260619 16:53:39.430519 21926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54d396362df54828a25574911fab8eb8: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:39.430848 21926 log.cc:824] T 00000000000000000000000000000000 P 54d396362df54828a25574911fab8eb8: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:39.431743 21926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54d396362df54828a25574911fab8eb8: No bootstrap required, opened a new log
I20260619 16:53:39.433701 21926 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54d396362df54828a25574911fab8eb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54d396362df54828a25574911fab8eb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40593 } }
I20260619 16:53:39.433882 21926 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54d396362df54828a25574911fab8eb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:39.433907 21926 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54d396362df54828a25574911fab8eb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54d396362df54828a25574911fab8eb8, State: Initialized, Role: FOLLOWER
I20260619 16:53:39.434072 21926 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54d396362df54828a25574911fab8eb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54d396362df54828a25574911fab8eb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40593 } }
I20260619 16:53:39.434154 21926 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54d396362df54828a25574911fab8eb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:53:39.434177 21926 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54d396362df54828a25574911fab8eb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:53:39.434271 21926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54d396362df54828a25574911fab8eb8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:39.434657 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 21858
I20260619 16:53:39.434787 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:53:39.434988 392 external_mini_cluster.cc:1428] Setting key 7b357fdf3af635556b331477dd42896a
I20260619 16:53:39.435182 21926 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54d396362df54828a25574911fab8eb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54d396362df54828a25574911fab8eb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40593 } }
I20260619 16:53:39.435295 21926 leader_election.cc:302] T 00000000000000000000000000000000 P 54d396362df54828a25574911fab8eb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54d396362df54828a25574911fab8eb8; no voters:
I20260619 16:53:39.435528 21926 leader_election.cc:288] T 00000000000000000000000000000000 P 54d396362df54828a25574911fab8eb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:53:39.435745 21931 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54d396362df54828a25574911fab8eb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:53:39.435853 21926 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54d396362df54828a25574911fab8eb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:53:39.435899 21931 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54d396362df54828a25574911fab8eb8 [term 1 LEADER]: Becoming Leader. State: Replica: 54d396362df54828a25574911fab8eb8, State: Running, Role: LEADER
I20260619 16:53:39.436038 21931 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54d396362df54828a25574911fab8eb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54d396362df54828a25574911fab8eb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40593 } }
I20260619 16:53:39.436439 21931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54d396362df54828a25574911fab8eb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54d396362df54828a25574911fab8eb8. Latest consensus state: current_term: 1 leader_uuid: "54d396362df54828a25574911fab8eb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54d396362df54828a25574911fab8eb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40593 } } }
I20260619 16:53:39.436514 21931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54d396362df54828a25574911fab8eb8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:39.436466 21933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54d396362df54828a25574911fab8eb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54d396362df54828a25574911fab8eb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54d396362df54828a25574911fab8eb8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40593 } } }
I20260619 16:53:39.436702 21933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54d396362df54828a25574911fab8eb8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:39.436834 21936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:53:39.437469 21936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:53:39.439514 21936 catalog_manager.cc:1347] Generated new cluster ID: 9b637274dfc849738846ece2254e0945
I20260619 16:53:39.439513 21932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:39.435605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.184:37103 (local address 127.0.0.1:40593)
0619 16:53:39.437617 (+ 2012us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:39.437629 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:39.437671 (+ 42us) server_negotiation.cc:407] Connection header received
0619 16:53:39.437824 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:39.437834 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:39.437872 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:39.438100 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:39.439020 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:39.439024 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:39.439024 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:39.439029 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:39.439076 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:39.439091 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:39.439158 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:39.439240 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 16:53:39.439289 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1939,"thread_start_us":119,"threads_started":1}
I20260619 16:53:39.439581 21936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:53:39.440487 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:53:39.443624 21932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:39.442305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.184:48607 (local address 127.0.0.1:40593)
0619 16:53:39.442346 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:39.442348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:39.442357 (+ 9us) server_negotiation.cc:407] Connection header received
0619 16:53:39.442425 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:39.442426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:39.442432 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:39.442480 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:39.443234 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:39.443236 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:39.443237 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:39.443241 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:39.443299 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:39.443305 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:39.443375 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:39.443423 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 16:53:39.443439 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 16:53:39.444680 21936 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:53:39.444854 21936 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:53:39.453044 21936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54d396362df54828a25574911fab8eb8: Generated new TSK 0
I20260619 16:53:39.453339 21936 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:53:39.461356 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 21858
2026-06-19T16:53:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1298 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:53:39 dist-test-slave-jsw9 krb5kdc[21957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:53:39 dist-test-slave-jsw9 krb5kdc[21957](info): set up 2 sockets
Jun 19 16:53:39 dist-test-slave-jsw9 krb5kdc[21957](info): commencing operation
krb5kdc: starting...
W20260619 16:53:40.438546 392 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:53:40 dist-test-slave-jsw9 krb5kdc[21957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:53:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:53:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:53:40.575042 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34945
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:53:40.675704 21970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:40.675992 21970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:40.676021 21970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:40.677404 21970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:53:40.677457 21970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:40.677471 21970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:40.677484 21970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:53:40.677495 21970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:53:40.679549 21970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34945
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:40.680094 21970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:40.680437 21970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:40.686765 21976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:40.686781 21978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:40.686813 21975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:40.687354 21970 server_base.cc:1034] running on GCE node
I20260619 16:53:40.687674 21970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:40.688074 21970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:40.689278 21970 hybrid_clock.cc:648] HybridClock initialized: now 1781888020689251 us; error 46 us; skew 500 ppm
Jun 19 16:53:40 dist-test-slave-jsw9 krb5kdc[21957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888020, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:40.697168 21970 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:53:40.697804 21970 webserver.cc:460] Webserver started at http://127.0.98.62:42079/ using document root <none> and password file <none>
I20260619 16:53:40.698098 21970 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:40.698159 21970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:40.698333 21970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:40.699653 21970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "6c069930985e4e1caa7b741601072a74"
format_stamp: "Formatted at 2026-06-19 16:53:40 on dist-test-slave-jsw9"
server_key: "fcb02a6d354d5477526a9b2c9f4b8af4"
server_key_iv: "09c2a644ce728d03be8f5339d049854f"
server_key_version: "encryptionkey@0"
I20260619 16:53:40.700161 21970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "6c069930985e4e1caa7b741601072a74"
format_stamp: "Formatted at 2026-06-19 16:53:40 on dist-test-slave-jsw9"
server_key: "fcb02a6d354d5477526a9b2c9f4b8af4"
server_key_iv: "09c2a644ce728d03be8f5339d049854f"
server_key_version: "encryptionkey@0"
I20260619 16:53:40.702029 21970 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:53:40.703011 21985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:40.703245 21970 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:53:40.703354 21970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "6c069930985e4e1caa7b741601072a74"
format_stamp: "Formatted at 2026-06-19 16:53:40 on dist-test-slave-jsw9"
server_key: "fcb02a6d354d5477526a9b2c9f4b8af4"
server_key_iv: "09c2a644ce728d03be8f5339d049854f"
server_key_version: "encryptionkey@0"
I20260619 16:53:40.703433 21970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:40.735474 21970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:40.737962 21970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:40.738123 21970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:40.742749 21970 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34945
I20260619 16:53:40.742796 22037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34945 every 8 connection(s)
I20260619 16:53:40.743193 21970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:53:40.744503 22038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:40.747520 22038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c069930985e4e1caa7b741601072a74: Bootstrap starting.
I20260619 16:53:40.748245 22038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c069930985e4e1caa7b741601072a74: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:40.748522 22038 log.cc:824] T 00000000000000000000000000000000 P 6c069930985e4e1caa7b741601072a74: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:40.749425 22038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c069930985e4e1caa7b741601072a74: No bootstrap required, opened a new log
I20260619 16:53:40.751511 22038 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c069930985e4e1caa7b741601072a74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c069930985e4e1caa7b741601072a74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } }
I20260619 16:53:40.751718 22038 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c069930985e4e1caa7b741601072a74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:40.751744 22038 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c069930985e4e1caa7b741601072a74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c069930985e4e1caa7b741601072a74, State: Initialized, Role: FOLLOWER
I20260619 16:53:40.751752 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 21970
I20260619 16:53:40.751883 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:53:40.751911 22038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c069930985e4e1caa7b741601072a74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c069930985e4e1caa7b741601072a74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } }
I20260619 16:53:40.751982 22038 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c069930985e4e1caa7b741601072a74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:53:40.752022 22038 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c069930985e4e1caa7b741601072a74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:53:40.752074 22038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c069930985e4e1caa7b741601072a74 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:40.752097 392 external_mini_cluster.cc:1428] Setting key d69a00471f677e5d7840b106b561a0de
I20260619 16:53:40.752894 22038 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c069930985e4e1caa7b741601072a74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c069930985e4e1caa7b741601072a74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } }
I20260619 16:53:40.753026 22038 leader_election.cc:302] T 00000000000000000000000000000000 P 6c069930985e4e1caa7b741601072a74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c069930985e4e1caa7b741601072a74; no voters:
I20260619 16:53:40.753235 22038 leader_election.cc:288] T 00000000000000000000000000000000 P 6c069930985e4e1caa7b741601072a74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:53:40.753274 22043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c069930985e4e1caa7b741601072a74 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:53:40.753389 22043 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c069930985e4e1caa7b741601072a74 [term 1 LEADER]: Becoming Leader. State: Replica: 6c069930985e4e1caa7b741601072a74, State: Running, Role: LEADER
I20260619 16:53:40.753556 22038 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c069930985e4e1caa7b741601072a74 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:53:40.753541 22043 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c069930985e4e1caa7b741601072a74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c069930985e4e1caa7b741601072a74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } }
I20260619 16:53:40.753994 22044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c069930985e4e1caa7b741601072a74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c069930985e4e1caa7b741601072a74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c069930985e4e1caa7b741601072a74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } }
I20260619 16:53:40.754084 22044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c069930985e4e1caa7b741601072a74 [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:40.754350 22045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c069930985e4e1caa7b741601072a74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c069930985e4e1caa7b741601072a74. Latest consensus state: current_term: 1 leader_uuid: "6c069930985e4e1caa7b741601072a74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c069930985e4e1caa7b741601072a74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34945 } } }
I20260619 16:53:40.754478 22045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c069930985e4e1caa7b741601072a74 [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:40.754766 22051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:53:40.755414 22051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:53:40.757485 22051 catalog_manager.cc:1347] Generated new cluster ID: aee8e02b846b4e6ba3f840758581faec
I20260619 16:53:40.757547 22051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 16:53:40 dist-test-slave-jsw9 krb5kdc[21957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:53:40.766927 22042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:40.753008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58493 (local address 127.0.98.62:34945)
0619 16:53:40.753217 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:40.753226 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:40.753260 (+ 34us) server_negotiation.cc:407] Connection header received
0619 16:53:40.753417 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:40.753427 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:40.753497 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:40.754555 (+ 1058us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:40.754727 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:40.756256 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:40.757281 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:40.758294 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:40.758419 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:40.762489 (+ 4070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:40.762497 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:40.762498 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:40.762518 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:40.764755 (+ 2237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:40.765351 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:40.765355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:40.765356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:40.765494 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:40.765754 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:40.765755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:40.765755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:40.766121 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:40.766342 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:40.766489 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 16:53:40.766612 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":113,"threads_started":1}
I20260619 16:53:40.768110 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:53:40.770076 22051 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:53:40.770797 22051 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:53:40.777518 22051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c069930985e4e1caa7b741601072a74: Generated new TSK 0
I20260619 16:53:40.777765 22051 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:53:40.778326 22042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:40.769933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41035 (local address 127.0.98.62:34945)
0619 16:53:40.769993 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:40.769995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:40.770404 (+ 409us) server_negotiation.cc:407] Connection header received
0619 16:53:40.770472 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:40.770472 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:40.770485 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:40.770540 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:40.771268 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:40.772038 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:40.773114 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:40.773434 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:40.773494 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:40.775168 (+ 1674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:40.775170 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:40.775170 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:40.775179 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:40.776796 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:40.777252 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:40.777253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:40.777254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:40.777328 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:40.777703 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:40.777704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:40.777704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:40.777851 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:40.777928 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:40.778133 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 16:53:40.778233 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 16:53:40.780684 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 21970
W20260619 16:53:40.785497 21955 connection.cc:535] client connection to 127.0.98.62:34945 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:34945): Connection reset by peer (error 104)
2026-06-19T16:53:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:53:40 dist-test-slave-jsw9 krb5kdc[22069](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:53:40 dist-test-slave-jsw9 krb5kdc[22069](info): set up 2 sockets
Jun 19 16:53:40 dist-test-slave-jsw9 krb5kdc[22069](info): commencing operation
krb5kdc: starting...
W20260619 16:53:41.759032 392 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:53:41 dist-test-slave-jsw9 krb5kdc[22069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:53:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:53:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:53:41.894250 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42529
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:53:41.989729 22082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:41.990039 22082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:41.990069 22082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:41.992048 22082 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:53:41.992146 22082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:41.992163 22082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:41.992177 22082 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:53:41.992188 22082 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:53:41.994323 22082 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42529
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:41.994669 22082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:41.994916 22082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:42.000530 22090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:42.000530 22088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:42.000530 22087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:42.000897 22082 server_base.cc:1034] running on GCE node
I20260619 16:53:42.001168 22082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:42.001520 22082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:42.002691 22082 hybrid_clock.cc:648] HybridClock initialized: now 1781888022002671 us; error 40 us; skew 500 ppm
Jun 19 16:53:42 dist-test-slave-jsw9 krb5kdc[22069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888022, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:42.006424 22082 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:53:42.007036 22082 webserver.cc:460] Webserver started at http://127.0.98.62:43813/ using document root <none> and password file <none>
I20260619 16:53:42.007289 22082 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:42.007344 22082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:42.007490 22082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:42.008800 22082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "c9ce39c335a04892a740bfbbb59e18b1"
format_stamp: "Formatted at 2026-06-19 16:53:42 on dist-test-slave-jsw9"
server_key: "2d691ce3c58a87d46ba4c1b9b181a787"
server_key_iv: "2e899190fbd9f04edcd1cd53e3f049b3"
server_key_version: "encryptionkey@0"
I20260619 16:53:42.009215 22082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "c9ce39c335a04892a740bfbbb59e18b1"
format_stamp: "Formatted at 2026-06-19 16:53:42 on dist-test-slave-jsw9"
server_key: "2d691ce3c58a87d46ba4c1b9b181a787"
server_key_iv: "2e899190fbd9f04edcd1cd53e3f049b3"
server_key_version: "encryptionkey@0"
I20260619 16:53:42.010867 22082 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 16:53:42.011791 22097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:42.012077 22082 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:53:42.012275 22082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "c9ce39c335a04892a740bfbbb59e18b1"
format_stamp: "Formatted at 2026-06-19 16:53:42 on dist-test-slave-jsw9"
server_key: "2d691ce3c58a87d46ba4c1b9b181a787"
server_key_iv: "2e899190fbd9f04edcd1cd53e3f049b3"
server_key_version: "encryptionkey@0"
I20260619 16:53:42.012357 22082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:42.033890 22082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:42.036438 22082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:42.036578 22082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:42.041164 22082 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42529
I20260619 16:53:42.041222 22149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42529 every 8 connection(s)
I20260619 16:53:42.041601 22082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:53:42.042713 22150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:42.045568 22150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9ce39c335a04892a740bfbbb59e18b1: Bootstrap starting.
I20260619 16:53:42.046348 22150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9ce39c335a04892a740bfbbb59e18b1: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:42.046658 22150 log.cc:824] T 00000000000000000000000000000000 P c9ce39c335a04892a740bfbbb59e18b1: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:42.047586 22150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9ce39c335a04892a740bfbbb59e18b1: No bootstrap required, opened a new log
I20260619 16:53:42.049489 22150 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9ce39c335a04892a740bfbbb59e18b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9ce39c335a04892a740bfbbb59e18b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42529 } }
I20260619 16:53:42.049712 22150 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9ce39c335a04892a740bfbbb59e18b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:42.049742 22150 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9ce39c335a04892a740bfbbb59e18b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9ce39c335a04892a740bfbbb59e18b1, State: Initialized, Role: FOLLOWER
I20260619 16:53:42.049918 22150 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9ce39c335a04892a740bfbbb59e18b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9ce39c335a04892a740bfbbb59e18b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42529 } }
I20260619 16:53:42.049999 22150 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9ce39c335a04892a740bfbbb59e18b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:53:42.050024 22150 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9ce39c335a04892a740bfbbb59e18b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:53:42.050060 22150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9ce39c335a04892a740bfbbb59e18b1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:42.050789 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 22082
I20260619 16:53:42.050913 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:53:42.050930 22150 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9ce39c335a04892a740bfbbb59e18b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9ce39c335a04892a740bfbbb59e18b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42529 } }
I20260619 16:53:42.051060 22150 leader_election.cc:302] T 00000000000000000000000000000000 P c9ce39c335a04892a740bfbbb59e18b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9ce39c335a04892a740bfbbb59e18b1; no voters:
I20260619 16:53:42.051129 392 external_mini_cluster.cc:1428] Setting key 074336c9efa0adfe418eeb939bab8dad
I20260619 16:53:42.051292 22150 leader_election.cc:288] T 00000000000000000000000000000000 P c9ce39c335a04892a740bfbbb59e18b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:53:42.051347 22153 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9ce39c335a04892a740bfbbb59e18b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:53:42.051549 22153 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9ce39c335a04892a740bfbbb59e18b1 [term 1 LEADER]: Becoming Leader. State: Replica: c9ce39c335a04892a740bfbbb59e18b1, State: Running, Role: LEADER
I20260619 16:53:42.051659 22150 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9ce39c335a04892a740bfbbb59e18b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:53:42.051714 22153 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9ce39c335a04892a740bfbbb59e18b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9ce39c335a04892a740bfbbb59e18b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42529 } }
I20260619 16:53:42.052361 22154 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9ce39c335a04892a740bfbbb59e18b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9ce39c335a04892a740bfbbb59e18b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9ce39c335a04892a740bfbbb59e18b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42529 } } }
I20260619 16:53:42.052526 22155 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9ce39c335a04892a740bfbbb59e18b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9ce39c335a04892a740bfbbb59e18b1. Latest consensus state: current_term: 1 leader_uuid: "c9ce39c335a04892a740bfbbb59e18b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9ce39c335a04892a740bfbbb59e18b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42529 } } }
I20260619 16:53:42.052599 22155 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9ce39c335a04892a740bfbbb59e18b1 [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:42.052794 22154 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9ce39c335a04892a740bfbbb59e18b1 [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:42.052971 22165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:53:42.053629 22165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:53:42.055787 22165 catalog_manager.cc:1347] Generated new cluster ID: a794094e12894e3aafeeaa4192fe60e4
I20260619 16:53:42.055855 22165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:53:42.058074 22157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:42.051724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37255 (local address 127.0.98.62:42529)
0619 16:53:42.051995 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:42.052006 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:42.052052 (+ 46us) server_negotiation.cc:407] Connection header received
0619 16:53:42.052203 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:42.052212 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:42.052290 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:42.052794 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:42.052947 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:42.054478 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:42.055562 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:42.056232 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:42.056360 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:42.057254 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:42.057259 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:42.057260 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:42.057513 (+ 253us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:42.057570 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:42.057586 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:42.057675 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:42.057782 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 16:53:42.057815 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":70,"threads_started":1}
I20260619 16:53:42.059392 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:53:42.066254 22157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:42.061209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60255 (local address 127.0.98.62:42529)
0619 16:53:42.061251 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:42.061253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:42.062398 (+ 1145us) server_negotiation.cc:407] Connection header received
0619 16:53:42.062474 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:42.062475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:42.062486 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:42.062535 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:42.062752 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:42.063528 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:42.064623 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:42.064967 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:42.065023 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:42.065940 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:42.065942 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:42.065943 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:42.065948 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:42.065967 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:42.065973 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:42.066024 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:42.066102 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 16:53:42.066118 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 16:53:42.091657 22165 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:53:42.092365 22165 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:53:42.095196 22165 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9ce39c335a04892a740bfbbb59e18b1: Generated new TSK 0
I20260619 16:53:42.095432 22165 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:53:42.107138 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 22082
W20260619 16:53:42.111599 22067 connection.cc:535] client connection to 127.0.98.62:42529 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:42529): Connection reset by peer (error 104)
2026-06-19T16:53:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:53:42 dist-test-slave-jsw9 krb5kdc[22182](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:53:42 dist-test-slave-jsw9 krb5kdc[22182](info): set up 2 sockets
Jun 19 16:53:42 dist-test-slave-jsw9 krb5kdc[22182](info): commencing operation
krb5kdc: starting...
W20260619 16:53:43.074528 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:53:43 dist-test-slave-jsw9 krb5kdc[22182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:53:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:53:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:53:43.206178 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45343
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:53:43.305256 22195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:43.305548 22195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:43.305576 22195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:43.306953 22195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:53:43.307016 22195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:43.307032 22195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:43.307044 22195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:53:43.307063 22195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:53:43.308792 22195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45343
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:43.309209 22195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:43.309523 22195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:43.315085 22203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:43.315357 22195 server_base.cc:1034] running on GCE node
W20260619 16:53:43.315085 22200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:43.315097 22201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:43.315891 22195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:43.316250 22195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:43.317444 22195 hybrid_clock.cc:648] HybridClock initialized: now 1781888023317418 us; error 39 us; skew 500 ppm
Jun 19 16:53:43 dist-test-slave-jsw9 krb5kdc[22182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:43.320777 22195 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:53:43.321604 22195 webserver.cc:460] Webserver started at http://127.0.98.62:34483/ using document root <none> and password file <none>
I20260619 16:53:43.321877 22195 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:43.321934 22195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:43.322332 22195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:43.324031 22195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "67a56372ac53427cb5662fe2a26fe10b"
format_stamp: "Formatted at 2026-06-19 16:53:43 on dist-test-slave-jsw9"
server_key: "97f39a6d95adbddf758483ceeb31059d"
server_key_iv: "e505eb1a248285502ebebf456fc3d31e"
server_key_version: "encryptionkey@0"
I20260619 16:53:43.324630 22195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "67a56372ac53427cb5662fe2a26fe10b"
format_stamp: "Formatted at 2026-06-19 16:53:43 on dist-test-slave-jsw9"
server_key: "97f39a6d95adbddf758483ceeb31059d"
server_key_iv: "e505eb1a248285502ebebf456fc3d31e"
server_key_version: "encryptionkey@0"
I20260619 16:53:43.326443 22195 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 16:53:43.327418 22210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:43.327661 22195 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:53:43.327780 22195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "67a56372ac53427cb5662fe2a26fe10b"
format_stamp: "Formatted at 2026-06-19 16:53:43 on dist-test-slave-jsw9"
server_key: "97f39a6d95adbddf758483ceeb31059d"
server_key_iv: "e505eb1a248285502ebebf456fc3d31e"
server_key_version: "encryptionkey@0"
I20260619 16:53:43.327862 22195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:43.331954 22195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:43.334432 22195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:43.334592 22195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:43.341203 22195 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45343
I20260619 16:53:43.341245 22262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45343 every 8 connection(s)
I20260619 16:53:43.341658 22195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:53:43.342577 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 22195
I20260619 16:53:43.342711 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:53:43.342932 392 external_mini_cluster.cc:1428] Setting key bdd9b047bf8797f55faea9e4c11b2fb7
I20260619 16:53:43.343081 22263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:43.346095 22266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:43.343598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37191 (local address 127.0.98.62:45343)
0619 16:53:43.343832 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:43.343841 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:43.343871 (+ 30us) server_negotiation.cc:407] Connection header received
0619 16:53:43.344436 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:43.344445 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:43.344477 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:43.344633 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:43.345649 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:43.345654 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:43.345655 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:43.345659 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:43.345707 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:43.345720 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:43.345798 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:43.345854 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 16:53:43.345902 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":95,"threads_started":1}
I20260619 16:53:43.347193 22263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67a56372ac53427cb5662fe2a26fe10b: Bootstrap starting.
I20260619 16:53:43.347245 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:53:43.348064 22263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67a56372ac53427cb5662fe2a26fe10b: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:43.348465 22263 log.cc:824] T 00000000000000000000000000000000 P 67a56372ac53427cb5662fe2a26fe10b: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:43.349323 22263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67a56372ac53427cb5662fe2a26fe10b: No bootstrap required, opened a new log
I20260619 16:53:43.350297 22266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:43.349053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44113 (local address 127.0.98.62:45343)
0619 16:53:43.349109 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:43.349110 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:43.349259 (+ 149us) server_negotiation.cc:407] Connection header received
0619 16:53:43.349326 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:43.349327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:43.349333 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:43.349382 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:43.350061 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:43.350062 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:43.350063 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:43.350067 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:43.350083 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:43.350088 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:43.350137 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:43.350215 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 16:53:43.350229 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 16:53:43.351786 22263 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67a56372ac53427cb5662fe2a26fe10b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a56372ac53427cb5662fe2a26fe10b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45343 } }
I20260619 16:53:43.351958 22263 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67a56372ac53427cb5662fe2a26fe10b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:43.351981 22263 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67a56372ac53427cb5662fe2a26fe10b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a56372ac53427cb5662fe2a26fe10b, State: Initialized, Role: FOLLOWER
I20260619 16:53:43.352097 22263 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67a56372ac53427cb5662fe2a26fe10b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a56372ac53427cb5662fe2a26fe10b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45343 } }
I20260619 16:53:43.352165 22263 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67a56372ac53427cb5662fe2a26fe10b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:53:43.352192 22263 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67a56372ac53427cb5662fe2a26fe10b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:53:43.352228 22263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67a56372ac53427cb5662fe2a26fe10b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:43.352886 22263 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67a56372ac53427cb5662fe2a26fe10b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a56372ac53427cb5662fe2a26fe10b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45343 } }
I20260619 16:53:43.353014 22263 leader_election.cc:302] T 00000000000000000000000000000000 P 67a56372ac53427cb5662fe2a26fe10b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67a56372ac53427cb5662fe2a26fe10b; no voters:
I20260619 16:53:43.353207 22263 leader_election.cc:288] T 00000000000000000000000000000000 P 67a56372ac53427cb5662fe2a26fe10b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:53:43.353364 22273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67a56372ac53427cb5662fe2a26fe10b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:53:43.353530 22273 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67a56372ac53427cb5662fe2a26fe10b [term 1 LEADER]: Becoming Leader. State: Replica: 67a56372ac53427cb5662fe2a26fe10b, State: Running, Role: LEADER
I20260619 16:53:43.353534 22263 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67a56372ac53427cb5662fe2a26fe10b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:53:43.353636 22273 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67a56372ac53427cb5662fe2a26fe10b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a56372ac53427cb5662fe2a26fe10b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45343 } }
I20260619 16:53:43.354056 22275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67a56372ac53427cb5662fe2a26fe10b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67a56372ac53427cb5662fe2a26fe10b. Latest consensus state: current_term: 1 leader_uuid: "67a56372ac53427cb5662fe2a26fe10b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a56372ac53427cb5662fe2a26fe10b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45343 } } }
I20260619 16:53:43.354251 22275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67a56372ac53427cb5662fe2a26fe10b [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:43.354595 22282 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:53:43.354688 22274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67a56372ac53427cb5662fe2a26fe10b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67a56372ac53427cb5662fe2a26fe10b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a56372ac53427cb5662fe2a26fe10b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45343 } } }
I20260619 16:53:43.354806 22274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67a56372ac53427cb5662fe2a26fe10b [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:43.355243 22282 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:53:43.356935 22282 catalog_manager.cc:1347] Generated new cluster ID: 2f819c2cc10f48dc83083454a071cae6
I20260619 16:53:43.356985 22282 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:53:43.372560 22282 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:53:43.372813 22282 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:53:43.376803 22282 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67a56372ac53427cb5662fe2a26fe10b: Generated new TSK 0
I20260619 16:53:43.377121 22282 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:53:43.384193 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 22195
2026-06-19T16:53:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1277 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:53:43 dist-test-slave-jsw9 krb5kdc[22294](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:53:43 dist-test-slave-jsw9 krb5kdc[22294](info): set up 2 sockets
Jun 19 16:53:43 dist-test-slave-jsw9 krb5kdc[22294](info): commencing operation
krb5kdc: starting...
W20260619 16:53:44.386610 392 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:53:44 dist-test-slave-jsw9 krb5kdc[22294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:53:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:53:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:53:44.530892 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43765
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:53:44.636019 22307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:44.636305 22307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:44.636343 22307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:44.637733 22307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:53:44.637791 22307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:44.637810 22307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:44.637826 22307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:53:44.637840 22307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:53:44.639621 22307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:44.639926 22307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:44.640189 22307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:44.646028 22315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:44.646031 22312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:44.646064 22307 server_base.cc:1034] running on GCE node
W20260619 16:53:44.646028 22313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:44.646533 22307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:44.646894 22307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:44.648092 22307 hybrid_clock.cc:648] HybridClock initialized: now 1781888024648058 us; error 50 us; skew 500 ppm
Jun 19 16:53:44 dist-test-slave-jsw9 krb5kdc[22294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:44.651290 22307 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:53:44.651857 22307 webserver.cc:460] Webserver started at http://127.0.98.62:39637/ using document root <none> and password file <none>
I20260619 16:53:44.652115 22307 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:44.652179 22307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:44.652308 22307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:44.653576 22307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "4cb3da4e1b914d29b202404e2173c6a8"
format_stamp: "Formatted at 2026-06-19 16:53:44 on dist-test-slave-jsw9"
server_key: "e3367e640139fe64adb18fa5f30bc907"
server_key_iv: "c973938427cb747eae67c26689ae7e9f"
server_key_version: "encryptionkey@0"
I20260619 16:53:44.653952 22307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "4cb3da4e1b914d29b202404e2173c6a8"
format_stamp: "Formatted at 2026-06-19 16:53:44 on dist-test-slave-jsw9"
server_key: "e3367e640139fe64adb18fa5f30bc907"
server_key_iv: "c973938427cb747eae67c26689ae7e9f"
server_key_version: "encryptionkey@0"
I20260619 16:53:44.655539 22307 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 16:53:44.656399 22322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:44.656602 22307 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:53:44.656703 22307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "4cb3da4e1b914d29b202404e2173c6a8"
format_stamp: "Formatted at 2026-06-19 16:53:44 on dist-test-slave-jsw9"
server_key: "e3367e640139fe64adb18fa5f30bc907"
server_key_iv: "c973938427cb747eae67c26689ae7e9f"
server_key_version: "encryptionkey@0"
I20260619 16:53:44.656781 22307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:44.677307 22307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:44.679894 22307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:44.680032 22307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:44.684496 22307 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43765
I20260619 16:53:44.684585 22374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43765 every 8 connection(s)
I20260619 16:53:44.684953 22307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:53:44.685989 22375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:44.687747 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 22307
I20260619 16:53:44.687861 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:53:44.688050 392 external_mini_cluster.cc:1428] Setting key c91c544e2b13d44e879ba58fd921e32d
I20260619 16:53:44.689003 22375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cb3da4e1b914d29b202404e2173c6a8: Bootstrap starting.
I20260619 16:53:44.689836 22375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cb3da4e1b914d29b202404e2173c6a8: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:44.690169 22375 log.cc:824] T 00000000000000000000000000000000 P 4cb3da4e1b914d29b202404e2173c6a8: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:44.690964 22375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cb3da4e1b914d29b202404e2173c6a8: No bootstrap required, opened a new log
I20260619 16:53:44.693776 22375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cb3da4e1b914d29b202404e2173c6a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb3da4e1b914d29b202404e2173c6a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43765 } }
I20260619 16:53:44.694010 22375 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4cb3da4e1b914d29b202404e2173c6a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:44.694054 22375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cb3da4e1b914d29b202404e2173c6a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cb3da4e1b914d29b202404e2173c6a8, State: Initialized, Role: FOLLOWER
I20260619 16:53:44.694231 22375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cb3da4e1b914d29b202404e2173c6a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb3da4e1b914d29b202404e2173c6a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43765 } }
I20260619 16:53:44.694368 22375 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4cb3da4e1b914d29b202404e2173c6a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:53:44.694416 22375 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cb3da4e1b914d29b202404e2173c6a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:53:44.694468 22375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cb3da4e1b914d29b202404e2173c6a8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:44.695468 22375 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cb3da4e1b914d29b202404e2173c6a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb3da4e1b914d29b202404e2173c6a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43765 } }
I20260619 16:53:44.695595 22375 leader_election.cc:302] T 00000000000000000000000000000000 P 4cb3da4e1b914d29b202404e2173c6a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cb3da4e1b914d29b202404e2173c6a8; no voters:
I20260619 16:53:44.696197 22375 leader_election.cc:288] T 00000000000000000000000000000000 P 4cb3da4e1b914d29b202404e2173c6a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:53:44.696241 22380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cb3da4e1b914d29b202404e2173c6a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:53:44.696434 22380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4cb3da4e1b914d29b202404e2173c6a8 [term 1 LEADER]: Becoming Leader. State: Replica: 4cb3da4e1b914d29b202404e2173c6a8, State: Running, Role: LEADER
I20260619 16:53:44.696630 22375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cb3da4e1b914d29b202404e2173c6a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:53:44.696625 22380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cb3da4e1b914d29b202404e2173c6a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb3da4e1b914d29b202404e2173c6a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43765 } }
I20260619 16:53:44.697151 22381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cb3da4e1b914d29b202404e2173c6a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cb3da4e1b914d29b202404e2173c6a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb3da4e1b914d29b202404e2173c6a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43765 } } }
Jun 19 16:53:44 dist-test-slave-jsw9 krb5kdc[22294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:53:44.697257 22381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cb3da4e1b914d29b202404e2173c6a8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:44.697502 22382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cb3da4e1b914d29b202404e2173c6a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cb3da4e1b914d29b202404e2173c6a8. Latest consensus state: current_term: 1 leader_uuid: "4cb3da4e1b914d29b202404e2173c6a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb3da4e1b914d29b202404e2173c6a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43765 } } }
I20260619 16:53:44.697566 22382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cb3da4e1b914d29b202404e2173c6a8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:44.697748 22388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:53:44.698827 22388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:53:44.700567 22388 catalog_manager.cc:1347] Generated new cluster ID: ff0097588ca14f25be88e1a7939b1c51
I20260619 16:53:44.700623 22388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:53:44.702168 22378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:44.688681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46011 (local address 127.0.98.62:43765)
0619 16:53:44.688900 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:44.688912 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:44.688951 (+ 39us) server_negotiation.cc:407] Connection header received
0619 16:53:44.689640 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:44.689651 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:44.689722 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:44.689907 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:44.690823 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:44.692474 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:44.693390 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:44.694071 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:44.694254 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:44.697913 (+ 3659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:44.697919 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:44.697919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:44.697936 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:44.700198 (+ 2262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:44.700727 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:44.700731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:44.700731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:44.700850 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:44.701110 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:44.701111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:44.701111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:44.701451 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:44.701665 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:44.701765 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 16:53:44.701939 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":91,"threads_started":1}
I20260619 16:53:44.703265 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:53:44.713667 22378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:44.704934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43333 (local address 127.0.98.62:43765)
0619 16:53:44.704984 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:44.704986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:44.705187 (+ 201us) server_negotiation.cc:407] Connection header received
0619 16:53:44.705239 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:44.705240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:44.705254 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:44.705328 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:44.706154 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:44.706986 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:44.708137 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:44.708460 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:44.708550 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:44.710240 (+ 1690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:44.710242 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:44.710243 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:44.710254 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:44.712066 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:44.712705 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:44.712707 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:44.712707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:44.712787 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:44.713022 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:44.713023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:44.713023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:44.713173 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:44.713288 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:44.713433 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 16:53:44.713547 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 16:53:44.713935 22388 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:53:44.714594 22388 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:53:44.719257 22388 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4cb3da4e1b914d29b202404e2173c6a8: Generated new TSK 0
I20260619 16:53:44.719552 22388 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:53:44.727376 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 22307
2026-06-19T16:53:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1343 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:53:44 dist-test-slave-jsw9 krb5kdc[22406](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:53:44 dist-test-slave-jsw9 krb5kdc[22406](info): set up 2 sockets
Jun 19 16:53:44 dist-test-slave-jsw9 krb5kdc[22406](info): commencing operation
krb5kdc: starting...
W20260619 16:53:45.710709 392 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:53:45 dist-test-slave-jsw9 krb5kdc[22406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:53:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:53:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:53:45.845815 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45255
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:53:45.940091 22419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:45.940385 22419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:45.940423 22419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:45.941833 22419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:53:45.941898 22419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:45.941918 22419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:45.941938 22419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:53:45.941954 22419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:53:45.943858 22419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:45.944206 22419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:45.944475 22419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:53:45.950027 22419 server_base.cc:1034] running on GCE node
W20260619 16:53:45.949991 22425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:45.949990 22427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:45.950110 22424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:45.950549 22419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:45.950906 22419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:45.952076 22419 hybrid_clock.cc:648] HybridClock initialized: now 1781888025952055 us; error 41 us; skew 500 ppm
Jun 19 16:53:45 dist-test-slave-jsw9 krb5kdc[22406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:45.962345 22419 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:53:45.962918 22419 webserver.cc:460] Webserver started at http://127.0.98.62:43713/ using document root <none> and password file <none>
I20260619 16:53:45.963172 22419 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:45.963230 22419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:45.963371 22419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:45.964805 22419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "316bdd2e3a6c4f77ab6399968d9b4bcd"
format_stamp: "Formatted at 2026-06-19 16:53:45 on dist-test-slave-jsw9"
server_key: "2c61dcd8f2e0a79bb9ec6b8363876f5e"
server_key_iv: "2faf54c7bac70f388398d3fac24347ba"
server_key_version: "encryptionkey@0"
I20260619 16:53:45.965273 22419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "316bdd2e3a6c4f77ab6399968d9b4bcd"
format_stamp: "Formatted at 2026-06-19 16:53:45 on dist-test-slave-jsw9"
server_key: "2c61dcd8f2e0a79bb9ec6b8363876f5e"
server_key_iv: "2faf54c7bac70f388398d3fac24347ba"
server_key_version: "encryptionkey@0"
I20260619 16:53:45.966895 22419 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260619 16:53:45.967841 22434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:45.968103 22419 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:53:45.968212 22419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "316bdd2e3a6c4f77ab6399968d9b4bcd"
format_stamp: "Formatted at 2026-06-19 16:53:45 on dist-test-slave-jsw9"
server_key: "2c61dcd8f2e0a79bb9ec6b8363876f5e"
server_key_iv: "2faf54c7bac70f388398d3fac24347ba"
server_key_version: "encryptionkey@0"
I20260619 16:53:45.968295 22419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:45.995599 22419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:45.998433 22419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:45.998626 22419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:46.003868 22419 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45255
I20260619 16:53:46.003901 22486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45255 every 8 connection(s)
I20260619 16:53:46.004424 22419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:53:46.005507 22487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:46.008785 22487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 316bdd2e3a6c4f77ab6399968d9b4bcd: Bootstrap starting.
I20260619 16:53:46.010170 22487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 316bdd2e3a6c4f77ab6399968d9b4bcd: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:46.010645 22487 log.cc:824] T 00000000000000000000000000000000 P 316bdd2e3a6c4f77ab6399968d9b4bcd: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:46.011715 22487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 316bdd2e3a6c4f77ab6399968d9b4bcd: No bootstrap required, opened a new log
I20260619 16:53:46.013032 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 22419
I20260619 16:53:46.013160 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:53:46.013350 392 external_mini_cluster.cc:1428] Setting key 064bf6f2d8ca8db193c641a949ad4574
I20260619 16:53:46.013903 22487 raft_consensus.cc:348] T 00000000000000000000000000000000 P 316bdd2e3a6c4f77ab6399968d9b4bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "316bdd2e3a6c4f77ab6399968d9b4bcd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45255 } }
I20260619 16:53:46.014159 22487 raft_consensus.cc:374] T 00000000000000000000000000000000 P 316bdd2e3a6c4f77ab6399968d9b4bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:46.014249 22487 raft_consensus.cc:729] T 00000000000000000000000000000000 P 316bdd2e3a6c4f77ab6399968d9b4bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316bdd2e3a6c4f77ab6399968d9b4bcd, State: Initialized, Role: FOLLOWER
I20260619 16:53:46.014407 22487 consensus_queue.cc:260] T 00000000000000000000000000000000 P 316bdd2e3a6c4f77ab6399968d9b4bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "316bdd2e3a6c4f77ab6399968d9b4bcd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45255 } }
I20260619 16:53:46.014513 22487 raft_consensus.cc:388] T 00000000000000000000000000000000 P 316bdd2e3a6c4f77ab6399968d9b4bcd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:53:46.014546 22487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 316bdd2e3a6c4f77ab6399968d9b4bcd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:53:46.014590 22487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 316bdd2e3a6c4f77ab6399968d9b4bcd [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:46.015604 22487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 316bdd2e3a6c4f77ab6399968d9b4bcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "316bdd2e3a6c4f77ab6399968d9b4bcd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45255 } }
I20260619 16:53:46.015733 22487 leader_election.cc:302] T 00000000000000000000000000000000 P 316bdd2e3a6c4f77ab6399968d9b4bcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 316bdd2e3a6c4f77ab6399968d9b4bcd; no voters:
I20260619 16:53:46.015939 22487 leader_election.cc:288] T 00000000000000000000000000000000 P 316bdd2e3a6c4f77ab6399968d9b4bcd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:53:46.016119 22492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 316bdd2e3a6c4f77ab6399968d9b4bcd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:53:46.016275 22487 sys_catalog.cc:564] T 00000000000000000000000000000000 P 316bdd2e3a6c4f77ab6399968d9b4bcd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:53:46.016388 22492 raft_consensus.cc:686] T 00000000000000000000000000000000 P 316bdd2e3a6c4f77ab6399968d9b4bcd [term 1 LEADER]: Becoming Leader. State: Replica: 316bdd2e3a6c4f77ab6399968d9b4bcd, State: Running, Role: LEADER
I20260619 16:53:46.016559 22492 consensus_queue.cc:237] T 00000000000000000000000000000000 P 316bdd2e3a6c4f77ab6399968d9b4bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "316bdd2e3a6c4f77ab6399968d9b4bcd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45255 } }
I20260619 16:53:46.017179 22494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 316bdd2e3a6c4f77ab6399968d9b4bcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 316bdd2e3a6c4f77ab6399968d9b4bcd. Latest consensus state: current_term: 1 leader_uuid: "316bdd2e3a6c4f77ab6399968d9b4bcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "316bdd2e3a6c4f77ab6399968d9b4bcd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45255 } } }
I20260619 16:53:46.017283 22494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 316bdd2e3a6c4f77ab6399968d9b4bcd [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:46.017328 22493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 316bdd2e3a6c4f77ab6399968d9b4bcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "316bdd2e3a6c4f77ab6399968d9b4bcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "316bdd2e3a6c4f77ab6399968d9b4bcd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45255 } } }
I20260619 16:53:46.017375 22493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 316bdd2e3a6c4f77ab6399968d9b4bcd [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:46.017570 22499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:53:46.018455 22499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:53:46.020507 22499 catalog_manager.cc:1347] Generated new cluster ID: 05ff18dc0df74388bb9711caf6f011bc
I20260619 16:53:46.020571 22499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:53:46.021602 22491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:46.014224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40485 (local address 127.0.98.62:45255)
0619 16:53:46.014518 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:46.014532 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:46.014579 (+ 47us) server_negotiation.cc:407] Connection header received
0619 16:53:46.014761 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:46.014770 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:46.014841 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:46.015025 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:46.015463 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:46.017110 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:46.018725 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:46.019329 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:46.019472 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:46.020772 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:46.020777 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:46.020778 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:46.021001 (+ 223us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:46.021040 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:46.021055 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:46.021121 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:46.021310 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 16:53:46.021403 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":172,"threads_started":1}
I20260619 16:53:46.022559 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:53:46.029736 22491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:46.026557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33409 (local address 127.0.98.62:45255)
0619 16:53:46.026611 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:46.026613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:46.026838 (+ 225us) server_negotiation.cc:407] Connection header received
0619 16:53:46.026898 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:46.026899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:46.026913 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:46.026965 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:46.027171 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:46.027710 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:46.028445 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:46.028676 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:46.028737 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:46.029454 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:46.029456 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:46.029456 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:46.029463 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:46.029481 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:46.029486 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:46.029541 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:46.029591 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 16:53:46.029630 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 16:53:46.060250 22499 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:53:46.060920 22499 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:53:46.064059 22499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 316bdd2e3a6c4f77ab6399968d9b4bcd: Generated new TSK 0
I20260619 16:53:46.064283 22499 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:53:46.069016 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 22419
2026-06-19T16:53:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1342 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:53:46 dist-test-slave-jsw9 krb5kdc[22518](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:53:46 dist-test-slave-jsw9 krb5kdc[22518](info): set up 2 sockets
Jun 19 16:53:46 dist-test-slave-jsw9 krb5kdc[22518](info): commencing operation
krb5kdc: starting...
W20260619 16:53:47.042575 392 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:53:47 dist-test-slave-jsw9 krb5kdc[22518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:53:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:53:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:53:47.174381 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46049
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:53:47.267563 22531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:47.267854 22531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:47.267894 22531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:47.269266 22531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:53:47.269318 22531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:47.269338 22531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:47.269356 22531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:53:47.269371 22531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:53:47.271096 22531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:47.271405 22531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:47.271657 22531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:47.277236 22539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:47.277226 22537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:47.277262 22536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:47.277443 22531 server_base.cc:1034] running on GCE node
I20260619 16:53:47.277709 22531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:47.278028 22531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:47.279206 22531 hybrid_clock.cc:648] HybridClock initialized: now 1781888027279183 us; error 34 us; skew 500 ppm
Jun 19 16:53:47 dist-test-slave-jsw9 krb5kdc[22518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:47.282226 22531 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:53:47.282783 22531 webserver.cc:460] Webserver started at http://127.0.98.62:41307/ using document root <none> and password file <none>
I20260619 16:53:47.283036 22531 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:47.283083 22531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:47.283200 22531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:47.284408 22531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "0cb8f4f781a847338f23f4c38909e7b8"
format_stamp: "Formatted at 2026-06-19 16:53:47 on dist-test-slave-jsw9"
server_key: "46d2f690469546e0d42a2880ff4ba626"
server_key_iv: "cc89d331f23a58b07d1e65fffb8eea5a"
server_key_version: "encryptionkey@0"
I20260619 16:53:47.284781 22531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "0cb8f4f781a847338f23f4c38909e7b8"
format_stamp: "Formatted at 2026-06-19 16:53:47 on dist-test-slave-jsw9"
server_key: "46d2f690469546e0d42a2880ff4ba626"
server_key_iv: "cc89d331f23a58b07d1e65fffb8eea5a"
server_key_version: "encryptionkey@0"
I20260619 16:53:47.286329 22531 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 16:53:47.287235 22546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:47.287467 22531 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:53:47.287564 22531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "0cb8f4f781a847338f23f4c38909e7b8"
format_stamp: "Formatted at 2026-06-19 16:53:47 on dist-test-slave-jsw9"
server_key: "46d2f690469546e0d42a2880ff4ba626"
server_key_iv: "cc89d331f23a58b07d1e65fffb8eea5a"
server_key_version: "encryptionkey@0"
I20260619 16:53:47.287645 22531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:47.291414 22531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:47.293879 22531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:47.294052 22531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:47.298568 22531 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46049
I20260619 16:53:47.298600 22598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46049 every 8 connection(s)
I20260619 16:53:47.298990 22531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:53:47.300040 22599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:47.300372 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 22531
I20260619 16:53:47.300472 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:53:47.300673 392 external_mini_cluster.cc:1428] Setting key 6cf8dcba6cbf6ccafe0002aad5618c0c
I20260619 16:53:47.303444 22599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cb8f4f781a847338f23f4c38909e7b8: Bootstrap starting.
I20260619 16:53:47.303534 22602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:47.301302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58853 (local address 127.0.98.62:46049)
0619 16:53:47.301522 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:47.301532 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:47.301574 (+ 42us) server_negotiation.cc:407] Connection header received
0619 16:53:47.302135 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:47.302145 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:47.302221 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:47.302385 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:47.303087 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:47.303091 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:47.303092 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:47.303096 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:47.303161 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:47.303177 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:47.303251 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:47.303284 (+ 33us) server_negotiation.cc:299] Negotiation successful
0619 16:53:47.303327 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":100,"threads_started":1}
I20260619 16:53:47.304222 22599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cb8f4f781a847338f23f4c38909e7b8: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:47.304560 22599 log.cc:824] T 00000000000000000000000000000000 P 0cb8f4f781a847338f23f4c38909e7b8: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:47.304585 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:53:47.305305 22599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cb8f4f781a847338f23f4c38909e7b8: No bootstrap required, opened a new log
I20260619 16:53:47.307243 22602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:47.306159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47429 (local address 127.0.98.62:46049)
0619 16:53:47.306227 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:47.306229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:47.306327 (+ 98us) server_negotiation.cc:407] Connection header received
0619 16:53:47.306372 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:47.306372 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:47.306377 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:47.306443 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:47.307005 (+ 562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:47.307007 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:47.307007 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:53:47.307011 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:47.307031 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:53:47.307037 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:47.307104 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:47.307148 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 16:53:47.307163 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 16:53:47.307667 22599 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cb8f4f781a847338f23f4c38909e7b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb8f4f781a847338f23f4c38909e7b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46049 } }
I20260619 16:53:47.307849 22599 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cb8f4f781a847338f23f4c38909e7b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:53:47.307883 22599 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cb8f4f781a847338f23f4c38909e7b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb8f4f781a847338f23f4c38909e7b8, State: Initialized, Role: FOLLOWER
I20260619 16:53:47.307997 22599 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cb8f4f781a847338f23f4c38909e7b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb8f4f781a847338f23f4c38909e7b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46049 } }
I20260619 16:53:47.308075 22599 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cb8f4f781a847338f23f4c38909e7b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:53:47.308100 22599 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cb8f4f781a847338f23f4c38909e7b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:53:47.308135 22599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cb8f4f781a847338f23f4c38909e7b8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:47.308768 22599 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cb8f4f781a847338f23f4c38909e7b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb8f4f781a847338f23f4c38909e7b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46049 } }
I20260619 16:53:47.308862 22599 leader_election.cc:302] T 00000000000000000000000000000000 P 0cb8f4f781a847338f23f4c38909e7b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cb8f4f781a847338f23f4c38909e7b8; no voters:
I20260619 16:53:47.309015 22599 leader_election.cc:288] T 00000000000000000000000000000000 P 0cb8f4f781a847338f23f4c38909e7b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:53:47.309070 22609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cb8f4f781a847338f23f4c38909e7b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:53:47.309249 22609 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cb8f4f781a847338f23f4c38909e7b8 [term 1 LEADER]: Becoming Leader. State: Replica: 0cb8f4f781a847338f23f4c38909e7b8, State: Running, Role: LEADER
I20260619 16:53:47.309265 22599 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cb8f4f781a847338f23f4c38909e7b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:53:47.309352 22609 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cb8f4f781a847338f23f4c38909e7b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb8f4f781a847338f23f4c38909e7b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46049 } }
I20260619 16:53:47.309762 22611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cb8f4f781a847338f23f4c38909e7b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cb8f4f781a847338f23f4c38909e7b8. Latest consensus state: current_term: 1 leader_uuid: "0cb8f4f781a847338f23f4c38909e7b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb8f4f781a847338f23f4c38909e7b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46049 } } }
I20260619 16:53:47.309863 22611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cb8f4f781a847338f23f4c38909e7b8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:47.310036 22610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cb8f4f781a847338f23f4c38909e7b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0cb8f4f781a847338f23f4c38909e7b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb8f4f781a847338f23f4c38909e7b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46049 } } }
I20260619 16:53:47.310103 22610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cb8f4f781a847338f23f4c38909e7b8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:47.310071 22617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:53:47.310758 22617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:53:47.312651 22617 catalog_manager.cc:1347] Generated new cluster ID: ddd2cb4162d84d00b022d41c8d149742
I20260619 16:53:47.312721 22617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:53:47.341228 22617 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:53:47.341459 22617 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:53:47.351146 22617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0cb8f4f781a847338f23f4c38909e7b8: Generated new TSK 0
I20260619 16:53:47.351384 22617 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:53:47.358932 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 22531
2026-06-19T16:53:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1291 ms)
[----------] 18 tests from AuthTokenIssuingTest (23487 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:53:47 dist-test-slave-jsw9 krb5kdc[22630](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:53:47 dist-test-slave-jsw9 krb5kdc[22630](info): set up 2 sockets
Jun 19 16:53:47 dist-test-slave-jsw9 krb5kdc[22630](info): commencing operation
krb5kdc: starting...
W20260619 16:53:48.330868 392 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:53:48 dist-test-slave-jsw9 krb5kdc[22630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:53:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:53:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:53:48.467334 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46517
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46517,127.0.98.61:43123,127.0.98.60:45877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:53:48.560218 22643 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 16:53:48.560514 22643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:48.560544 22643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:48.560573 22643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:48.561915 22643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:53:48.561973 22643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:48.561995 22643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:48.562011 22643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:53:48.562026 22643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:53:48.563771 22643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46517,127.0.98.61:43123,127.0.98.60:45877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:48.564075 22643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:48.564327 22643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:53:48.569630 22643 server_base.cc:1034] running on GCE node
W20260619 16:53:48.569571 22649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:48.569567 22651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:48.569578 22648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:48.570073 22643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:48.570443 22643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:48.571591 22643 hybrid_clock.cc:648] HybridClock initialized: now 1781888028571566 us; error 42 us; skew 500 ppm
Jun 19 16:53:48 dist-test-slave-jsw9 krb5kdc[22630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:48.574450 22643 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:53:48.574934 22643 webserver.cc:460] Webserver started at http://127.0.98.62:41169/ using document root <none> and password file <none>
I20260619 16:53:48.575171 22643 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:48.575222 22643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:48.575352 22643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:48.576534 22643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "25f2c9fcad824daa897e9095362eaff0"
format_stamp: "Formatted at 2026-06-19 16:53:48 on dist-test-slave-jsw9"
server_key: "e650719504efec0a95a37ec59b3d4e23"
server_key_iv: "8f039751604e6242005114722cc3150e"
server_key_version: "encryptionkey@0"
I20260619 16:53:48.576892 22643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "25f2c9fcad824daa897e9095362eaff0"
format_stamp: "Formatted at 2026-06-19 16:53:48 on dist-test-slave-jsw9"
server_key: "e650719504efec0a95a37ec59b3d4e23"
server_key_iv: "8f039751604e6242005114722cc3150e"
server_key_version: "encryptionkey@0"
I20260619 16:53:48.578357 22643 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 16:53:48.580360 22658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:48.580575 22643 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:53:48.580679 22643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "25f2c9fcad824daa897e9095362eaff0"
format_stamp: "Formatted at 2026-06-19 16:53:48 on dist-test-slave-jsw9"
server_key: "e650719504efec0a95a37ec59b3d4e23"
server_key_iv: "8f039751604e6242005114722cc3150e"
server_key_version: "encryptionkey@0"
I20260619 16:53:48.580770 22643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:48.602357 22643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:48.605077 22643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:48.605206 22643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:48.610247 22643 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46517
I20260619 16:53:48.610322 22710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46517 every 8 connection(s)
I20260619 16:53:48.610687 22643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:53:48.611747 22711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:48.613453 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 22643
I20260619 16:53:48.613583 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:53:48.613766 392 external_mini_cluster.cc:1428] Setting key cc7a5bbf2ec5c620bf8954efb1176409
I20260619 16:53:48.613803 22711 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46517 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:53:48 dist-test-slave-jsw9 krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260619 16:53:48.624989 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.614766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.0.1:41938)
0619 16:53:48.614989 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:48.615038 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:48.615714 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:48.616007 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:48.616010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:48.616322 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:48.616535 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:48.616541 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.617749 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.617751 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:48.618517 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:48.618519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.619035 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.619037 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:48.619104 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:48.619673 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:53:48.619684 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:48.621264 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:48.623128 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:48.623135 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:48.623137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:48.623376 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:48.623645 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:48.623645 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:48.623646 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:48.623730 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:48.624196 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:48.624203 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:48.624286 (+ 83us) client_negotiation.cc:769] Sending connection context
0619 16:53:48.624391 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 16:53:48.624726 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":171,"thread_start_us":108,"threads_started":1}
I20260619 16:53:48.625036 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.614783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41938 (local address 127.0.98.62:46517)
0619 16:53:48.614998 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:48.615008 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:48.615140 (+ 132us) server_negotiation.cc:407] Connection header received
0619 16:53:48.615879 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:48.615882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:48.615905 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:48.615990 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:48.616672 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.617608 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.618593 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.618938 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.619043 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:48.621373 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:48.621382 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:48.621382 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:48.621389 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:48.623040 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:48.623468 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:48.623471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:48.623472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:48.623562 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:48.623806 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:48.623807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:48.623807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:48.624089 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:48.624187 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:48.624462 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 16:53:48.624644 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":181,"spinlock_wait_cycles":11648,"thread_start_us":110,"threads_started":1}
I20260619 16:53:48.625831 22711 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:53:48.626310 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.626147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43123 (local address 127.0.0.1:40134)
0619 16:53:48.626238 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:48.626286 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260619 16:53:48.626448 22662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43123: connect: Connection refused (error 111)
I20260619 16:53:48.626875 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.626812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43123 (local address 127.0.0.1:40136)
0619 16:53:48.626836 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:48.626858 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260619 16:53:48.626967 22711 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43123: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43123: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 16:53:48.641192 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43123
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46517,127.0.98.61:43123,127.0.98.60:45877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:53:48.676258 22711 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } attempt: 1
I20260619 16:53:48.677212 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.677021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43123 (local address 127.0.0.1:40144)
0619 16:53:48.677123 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:48.677164 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 16:53:48.677918 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.677746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43123 (local address 127.0.0.1:40160)
0619 16:53:48.677822 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:48.677874 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 16:53:48.678037 22711 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43123: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43123: connect: Connection refused (error 111)
W20260619 16:53:48.733636 22718 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 16:53:48.733929 22718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:48.733950 22718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:48.733973 22718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:48.735455 22718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:53:48.735514 22718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:48.735527 22718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:48.735539 22718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:53:48.735551 22718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:53:48.737284 22718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46517,127.0.98.61:43123,127.0.98.60:45877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:48.737584 22718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:48.737839 22718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:48.743086 22726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:48.743088 22723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:48.743081 22724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:48.743472 22718 server_base.cc:1034] running on GCE node
I20260619 16:53:48.743722 22718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:48.744024 22718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:48.745219 22718 hybrid_clock.cc:648] HybridClock initialized: now 1781888028745197 us; error 33 us; skew 500 ppm
I20260619 16:53:48.746240 22711 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } attempt: 2
I20260619 16:53:48.746721 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.746586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43123 (local address 127.0.0.1:40174)
0619 16:53:48.746651 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:48.746686 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 16:53:48.747136 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.747008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43123 (local address 127.0.0.1:40188)
0619 16:53:48.747073 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:48.747107 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 16:53:48.747249 22711 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43123: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43123: connect: Connection refused (error 111)
Jun 19 16:53:48 dist-test-slave-jsw9 krb5kdc[22630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:48.748678 22718 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 16:53:48.749228 22718 webserver.cc:460] Webserver started at http://127.0.98.61:39477/ using document root <none> and password file <none>
I20260619 16:53:48.749480 22718 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:48.749534 22718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:48.749675 22718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:48.750958 22718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-1/data/instance:
uuid: "6d0102ef76554921ad64934924015fc1"
format_stamp: "Formatted at 2026-06-19 16:53:48 on dist-test-slave-jsw9"
server_key: "f67a67da290b869b4cf6721cd5f27ee1"
server_key_iv: "447d223bbb47dde844a63164289ac34e"
server_key_version: "encryptionkey@0"
I20260619 16:53:48.751348 22718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-1/wal/instance:
uuid: "6d0102ef76554921ad64934924015fc1"
format_stamp: "Formatted at 2026-06-19 16:53:48 on dist-test-slave-jsw9"
server_key: "f67a67da290b869b4cf6721cd5f27ee1"
server_key_iv: "447d223bbb47dde844a63164289ac34e"
server_key_version: "encryptionkey@0"
I20260619 16:53:48.752866 22718 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 16:53:48.753736 22733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:48.753952 22718 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:53:48.754047 22718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-1/wal
uuid: "6d0102ef76554921ad64934924015fc1"
format_stamp: "Formatted at 2026-06-19 16:53:48 on dist-test-slave-jsw9"
server_key: "f67a67da290b869b4cf6721cd5f27ee1"
server_key_iv: "447d223bbb47dde844a63164289ac34e"
server_key_version: "encryptionkey@0"
I20260619 16:53:48.754120 22718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:48.768201 22718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:48.770776 22718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:48.770920 22718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:48.775735 22718 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43123
I20260619 16:53:48.775765 22785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43123 every 8 connection(s)
I20260619 16:53:48.776185 22718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-1/data/info.pb
I20260619 16:53:48.777285 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 22718
I20260619 16:53:48.777392 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-1/wal/instance
I20260619 16:53:48.777449 22786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:48.777596 392 external_mini_cluster.cc:1428] Setting key dc504df00321acb166dc5836ffd854cb
I20260619 16:53:48.780120 22786 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46517 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 19 16:53:48 dist-test-slave-jsw9 krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:53:48.794821 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.781066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41942 (local address 127.0.98.62:46517)
0619 16:53:48.781175 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:48.781178 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:48.781398 (+ 220us) server_negotiation.cc:407] Connection header received
0619 16:53:48.782222 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:48.782224 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:48.782244 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:48.782318 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:48.783050 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.783855 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.785097 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.785446 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.785527 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:48.790967 (+ 5440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:48.790970 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:48.790970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:48.790982 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:48.792660 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:48.793413 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:48.793415 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:48.793416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:48.793506 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:48.793883 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:48.793884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:48.793884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:48.794049 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:48.794093 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:48.794585 (+ 492us) server_negotiation.cc:299] Negotiation successful
0619 16:53:48.794701 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 16:53:48.795542 22789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.781037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.0.1:41942)
0619 16:53:48.781259 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:48.781304 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:48.781973 (+ 669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:48.782391 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:48.782395 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:48.782709 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:48.782951 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:48.782955 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.783985 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.783986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:48.785004 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:48.785008 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.785583 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.785584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:48.785827 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:48.786462 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:53:48.786485 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:48.790818 (+ 4333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:48.792790 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:48.792806 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:48.792811 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:48.793175 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:48.793602 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:48.793603 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:48.793603 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:48.793799 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:48.794152 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:48.794162 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:48.794406 (+ 244us) client_negotiation.cc:769] Sending connection context
0619 16:53:48.795000 (+ 594us) client_negotiation.cc:240] Negotiation successful
0619 16:53:48.795254 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":107,"threads_started":1}
I20260619 16:53:48.796212 22786 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:53:48 dist-test-slave-jsw9 krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 16:53:48.805553 22789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.796528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43123 (local address 127.0.0.1:40198)
0619 16:53:48.796759 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:48.796771 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:48.796828 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:48.796985 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:48.796986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:48.797148 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:48.797212 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:48.797214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.798267 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.798268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:48.799011 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:48.799020 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.799858 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.799859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:48.799941 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:48.800424 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:53:48.800429 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:48.802460 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:48.804450 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:48.804451 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:48.804452 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:48.804632 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:48.804826 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:48.804827 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:48.804828 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:48.804882 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:48.805178 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:48.805179 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:48.805222 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 16:53:48.805286 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 16:53:48.805422 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211}
I20260619 16:53:48.805627 22791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.796600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40198 (local address 127.0.98.61:43123)
0619 16:53:48.796777 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:48.796782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:48.796825 (+ 43us) server_negotiation.cc:407] Connection header received
0619 16:53:48.796882 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:48.796886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:48.796900 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:48.796975 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:48.797280 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.798157 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.799195 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.799742 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.799844 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:48.802578 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:48.802590 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:48.802591 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:48.802598 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:48.804354 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:48.804712 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:48.804715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:48.804716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:48.804769 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:48.804930 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:48.804931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:48.804932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:48.805113 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:48.805194 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:48.805326 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 16:53:48.805499 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":148,"thread_start_us":99,"threads_started":1}
I20260619 16:53:48.806522 22786 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45877 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:53:48.806979 22789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.806803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45877 (local address 127.0.0.1:37088)
0619 16:53:48.806876 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:48.806947 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 16:53:48.807248 22735 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45877: connect: Connection refused (error 111)
I20260619 16:53:48.807871 22789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.807727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45877 (local address 127.0.0.1:37104)
0619 16:53:48.807790 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:48.807838 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 16:53:48.807996 22786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45877: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45877: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 16:53:48.810631 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45877
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46517,127.0.98.61:43123,127.0.98.60:45877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:53:48.838411 22711 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } attempt: 3
Jun 19 16:53:48 dist-test-slave-jsw9 krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 16:53:48.845927 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.838880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43123 (local address 127.0.0.1:40200)
0619 16:53:48.838956 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:48.838969 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:48.839062 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:48.839207 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:48.839207 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:48.839385 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:48.839462 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:48.839464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.840184 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.840185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:48.840726 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:48.840728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.841092 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.841093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:48.841174 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:48.841655 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:53:48.841660 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:48.843251 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:48.844785 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:48.844786 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:48.844787 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:48.844963 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:48.845263 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:48.845264 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:48.845265 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:48.845334 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:48.845625 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:48.845626 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:48.845672 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 16:53:48.845724 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 16:53:48.845841 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 16:53:48.845952 22791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.838971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40200 (local address 127.0.98.61:43123)
0619 16:53:48.839041 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:48.839043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:48.839055 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:53:48.839135 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:48.839136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:48.839150 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:48.839200 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:48.839557 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.840100 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.840803 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.841030 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.841090 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:48.843343 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:48.843345 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:48.843346 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:48.843356 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:48.844707 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:48.845063 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:48.845064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:48.845065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:48.845140 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:48.845401 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:48.845402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:48.845402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:48.845563 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:48.845637 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:48.845763 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 16:53:48.845853 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 16:53:48.846439 22711 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45877 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:53:48.846905 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.846711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45877 (local address 127.0.0.1:37110)
0619 16:53:48.846762 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:48.846803 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 16:53:48.847321 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.847227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45877 (local address 127.0.0.1:37120)
0619 16:53:48.847266 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:48.847299 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 16:53:48.847409 22711 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45877: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45877: connect: Connection refused (error 111)
I20260619 16:53:48.857254 22786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45877 } attempt: 1
I20260619 16:53:48.857816 22789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.857674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45877 (local address 127.0.0.1:37134)
0619 16:53:48.857735 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:48.857776 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 16:53:48.858302 22789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.858126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45877 (local address 127.0.0.1:37138)
0619 16:53:48.858230 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:48.858269 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260619 16:53:48.858404 22786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45877: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45877: connect: Connection refused (error 111)
I20260619 16:53:48.878557 22711 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45877 } attempt: 1
I20260619 16:53:48.879243 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.879078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45877 (local address 127.0.0.1:37140)
0619 16:53:48.879147 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:48.879201 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 16:53:48.879698 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.879563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45877 (local address 127.0.0.1:37152)
0619 16:53:48.879634 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:48.879673 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 16:53:48.879814 22711 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45877: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45877: connect: Connection refused (error 111)
W20260619 16:53:48.905586 22793 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 16:53:48.905870 22793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:48.905889 22793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:48.905911 22793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:48.907253 22793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:53:48.907305 22793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:48.907317 22793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:48.907330 22793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:53:48.907339 22793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:53:48.908924 22793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46517,127.0.98.61:43123,127.0.98.60:45877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:48.909176 22793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:48.909392 22793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:48.914753 22801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:48.914753 22799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:48.914783 22798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:48.915079 22793 server_base.cc:1034] running on GCE node
I20260619 16:53:48.915331 22793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:48.915650 22793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:48.916826 22793 hybrid_clock.cc:648] HybridClock initialized: now 1781888028916802 us; error 31 us; skew 500 ppm
Jun 19 16:53:48 dist-test-slave-jsw9 krb5kdc[22630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:48.919615 22793 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 16:53:48.920248 22793 webserver.cc:460] Webserver started at http://127.0.98.60:36943/ using document root <none> and password file <none>
I20260619 16:53:48.920490 22793 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:48.920544 22793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:48.920686 22793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:48.921905 22793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-2/data/instance:
uuid: "ba6efe458efb4c488d47cd11e8f782bc"
format_stamp: "Formatted at 2026-06-19 16:53:48 on dist-test-slave-jsw9"
server_key: "e901c0253449dd4d0c122104f9893695"
server_key_iv: "2bf0575a5bf18dd6eae9ed2b4d76988d"
server_key_version: "encryptionkey@0"
I20260619 16:53:48.922320 22793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-2/wal/instance:
uuid: "ba6efe458efb4c488d47cd11e8f782bc"
format_stamp: "Formatted at 2026-06-19 16:53:48 on dist-test-slave-jsw9"
server_key: "e901c0253449dd4d0c122104f9893695"
server_key_iv: "2bf0575a5bf18dd6eae9ed2b4d76988d"
server_key_version: "encryptionkey@0"
I20260619 16:53:48.923759 22793 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:53:48.924566 22808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:48.924732 22793 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:48.924822 22793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-2/wal
uuid: "ba6efe458efb4c488d47cd11e8f782bc"
format_stamp: "Formatted at 2026-06-19 16:53:48 on dist-test-slave-jsw9"
server_key: "e901c0253449dd4d0c122104f9893695"
server_key_iv: "2bf0575a5bf18dd6eae9ed2b4d76988d"
server_key_version: "encryptionkey@0"
I20260619 16:53:48.924897 22793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:48.926545 22786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45877 } attempt: 2
I20260619 16:53:48.927024 22789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.926864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45877 (local address 127.0.0.1:37162)
0619 16:53:48.926932 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:48.926979 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 16:53:48.927438 22789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.927338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45877 (local address 127.0.0.1:37174)
0619 16:53:48.927382 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:48.927411 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 16:53:48.927574 22786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45877: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45877: connect: Connection refused (error 111)
I20260619 16:53:48.934988 22793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:48.937467 22793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:48.937628 22793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:48.942166 22793 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45877
I20260619 16:53:48.942227 22860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45877 every 8 connection(s)
I20260619 16:53:48.942626 22793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-2/data/info.pb
I20260619 16:53:48.943753 22861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:48.945562 22861 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46517 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:53:48.946780 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 22793
I20260619 16:53:48.946882 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781887741903642-392-0/minicluster-data/master-2/wal/instance
I20260619 16:53:48.947049 392 external_mini_cluster.cc:1428] Setting key c32bea0f1e63f76726380b2ed3a31cbf
Jun 19 16:53:48 dist-test-slave-jsw9 krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 19 16:53:48 dist-test-slave-jsw9 krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 16:53:48.954982 22711 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45877 } attempt: 2
I20260619 16:53:48.956719 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.946475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41958 (local address 127.0.98.62:46517)
0619 16:53:48.946561 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:48.946563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:48.946764 (+ 201us) server_negotiation.cc:407] Connection header received
0619 16:53:48.947423 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:48.947424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:48.947438 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:48.947494 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:48.948434 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.949165 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.950440 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.950653 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.950715 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:48.953257 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:48.953258 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:48.953259 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:48.953268 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:48.954852 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:48.955622 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:48.955624 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:48.955625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:48.955694 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:48.955986 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:48.955987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:48.955987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:48.956122 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:48.956157 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:48.956528 (+ 371us) server_negotiation.cc:299] Negotiation successful
0619 16:53:48.956618 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 16:53:48.957301 22865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.947488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37188 (local address 127.0.98.60:45877)
0619 16:53:48.947651 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:48.947656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:48.947705 (+ 49us) server_negotiation.cc:407] Connection header received
0619 16:53:48.947775 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:48.947776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:48.947797 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:48.947870 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:48.948637 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.949784 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.950736 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.951052 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.951115 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:48.953674 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:48.953682 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:48.953683 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:48.953689 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:48.955665 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:48.956057 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:48.956060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:48.956061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:48.956134 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:48.956325 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:48.956325 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:48.956326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:48.956491 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:48.956646 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:48.956876 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 16:53:48.956947 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":108,"threads_started":1}
I20260619 16:53:48.957301 22863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.946418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46517 (local address 127.0.0.1:41958)
0619 16:53:48.946621 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:48.946670 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:48.947284 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:48.947643 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:48.947649 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:48.948068 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:48.948333 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:48.948339 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.949273 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.949274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:48.950360 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:48.950364 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.950752 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.950753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:48.950879 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:48.951410 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:53:48.951422 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:48.953152 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:48.954973 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:48.954985 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:48.954987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:48.955550 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:48.955764 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:48.955765 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:48.955765 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:48.955908 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:48.956203 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:48.956210 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:48.956398 (+ 188us) client_negotiation.cc:769] Sending connection context
0619 16:53:48.956494 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 16:53:48.956743 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":36,"thread_start_us":85,"threads_started":1}
I20260619 16:53:48.958173 22861 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:53:48 dist-test-slave-jsw9 krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 16:53:48 dist-test-slave-jsw9 krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 19 16:53:48 dist-test-slave-jsw9 krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:53:48.963923 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.955316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45877 (local address 127.0.0.1:37190)
0619 16:53:48.955363 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:48.955375 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:48.955423 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:48.955729 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:48.955730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:48.955867 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:48.955918 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:48.955920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.956771 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.956772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:48.957683 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:48.957686 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.958294 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.958295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:48.958360 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:48.958787 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:53:48.958791 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:48.960268 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:48.962666 (+ 2398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:48.962667 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:48.962667 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:48.962836 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:48.963047 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:48.963048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:48.963049 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:48.963112 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:48.963606 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:48.963607 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:48.963648 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 16:53:48.963706 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 16:53:48.963811 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 16:53:48.963955 22866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.955373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37190 (local address 127.0.98.60:45877)
0619 16:53:48.955604 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:48.955606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:48.955614 (+ 8us) server_negotiation.cc:407] Connection header received
0619 16:53:48.955632 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:48.955633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:48.955644 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:48.955712 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:48.955981 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.956685 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.957889 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.958173 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.958289 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:48.960342 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:48.960343 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:48.960344 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:48.960351 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:48.962593 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:48.962895 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:48.962896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:48.962897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:48.962955 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:48.963164 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:48.963165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:48.963165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:48.963277 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:48.963342 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:48.963745 (+ 403us) server_negotiation.cc:299] Negotiation successful
0619 16:53:48.963817 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":131,"threads_started":1}
I20260619 16:53:48.965981 22711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25f2c9fcad824daa897e9095362eaff0: Bootstrap starting.
I20260619 16:53:48.966557 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.958277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41962 (local address 127.0.98.62:46517)
0619 16:53:48.958373 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:48.958375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:48.958472 (+ 97us) server_negotiation.cc:407] Connection header received
0619 16:53:48.958529 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:48.958529 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:48.958539 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:48.958583 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:48.959333 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.960031 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.960886 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.961155 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.961196 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:48.963938 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:48.963939 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:48.963939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:48.963946 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:48.965367 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:48.965678 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:48.965679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:48.965679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:48.965735 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:48.965897 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:48.965898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:48.965898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:48.966009 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:48.966180 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:48.966387 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 16:53:48.966465 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 16:53:48.966949 22711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25f2c9fcad824daa897e9095362eaff0: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:48.967325 22711 log.cc:824] T 00000000000000000000000000000000 P 25f2c9fcad824daa897e9095362eaff0: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:48.968391 22711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25f2c9fcad824daa897e9095362eaff0: No bootstrap required, opened a new log
I20260619 16:53:48.968397 22791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.958967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40214 (local address 127.0.98.61:43123)
0619 16:53:48.959030 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:48.959032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:48.959169 (+ 137us) server_negotiation.cc:407] Connection header received
0619 16:53:48.959264 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:48.959265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:48.959276 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:48.959306 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:48.959629 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.960134 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.960997 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.961284 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.961346 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:48.963361 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:48.963363 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:48.963363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:48.963371 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:48.966854 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:48.967430 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:48.967432 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:48.967432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:48.967509 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:48.967740 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:48.967741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:48.967741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:48.967883 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:48.967959 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:48.968214 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 16:53:48.968303 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 16:53:48.968685 22863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.959055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43123 (local address 127.0.0.1:40214)
0619 16:53:48.959116 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:48.959126 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:48.959192 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:48.959372 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:48.959373 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:48.959504 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:48.959552 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:48.959553 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.960203 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.960204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:48.960913 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:48.960916 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.961352 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.961353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:48.961433 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:48.961845 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:53:48.961849 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:48.963259 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:48.966968 (+ 3709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:48.966969 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:48.966969 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:48.967252 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:48.967613 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:48.967614 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:48.967615 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:48.967689 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:48.967988 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:48.967989 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:48.968085 (+ 96us) client_negotiation.cc:769] Sending connection context
0619 16:53:48.968473 (+ 388us) client_negotiation.cc:240] Negotiation successful
0619 16:53:48.968595 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 16:53:48.970649 22711 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25f2c9fcad824daa897e9095362eaff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25f2c9fcad824daa897e9095362eaff0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46517 } } peers { permanent_uuid: "6d0102ef76554921ad64934924015fc1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } } peers { permanent_uuid: "ba6efe458efb4c488d47cd11e8f782bc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45877 } }
I20260619 16:53:48.970835 22711 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25f2c9fcad824daa897e9095362eaff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25f2c9fcad824daa897e9095362eaff0, State: Initialized, Role: FOLLOWER
I20260619 16:53:48.970943 22711 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25f2c9fcad824daa897e9095362eaff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25f2c9fcad824daa897e9095362eaff0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46517 } } peers { permanent_uuid: "6d0102ef76554921ad64934924015fc1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } } peers { permanent_uuid: "ba6efe458efb4c488d47cd11e8f782bc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45877 } }
I20260619 16:53:48.971359 22711 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25f2c9fcad824daa897e9095362eaff0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:53:48.971326 22869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25f2c9fcad824daa897e9095362eaff0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25f2c9fcad824daa897e9095362eaff0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46517 } } peers { permanent_uuid: "6d0102ef76554921ad64934924015fc1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } } peers { permanent_uuid: "ba6efe458efb4c488d47cd11e8f782bc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45877 } } }
I20260619 16:53:48.971405 22869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25f2c9fcad824daa897e9095362eaff0 [sys.catalog]: This master's current role is: FOLLOWER
Jun 19 16:53:48 dist-test-slave-jsw9 krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 16:53:48.973053 22861 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45877 } has no permanent_uuid. Determining permanent_uuid...
W20260619 16:53:48.973233 22880 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 25f2c9fcad824daa897e9095362eaff0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:53:48.973289 22880 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:53:48.974962 22643 master_runner.cc:186] Error getting master registration for 127.0.98.61:43123: OK, instance_id { permanent_uuid: "6d0102ef76554921ad64934924015fc1" instance_seqno: 1781888028774325 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 16:53:48.978111 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.967556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40216 (local address 127.0.98.61:43123)
0619 16:53:48.967738 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:48.967740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:48.967747 (+ 7us) server_negotiation.cc:407] Connection header received
0619 16:53:48.967768 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:48.967769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:48.967780 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:48.967865 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:48.968482 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.969614 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.970392 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.970694 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.970771 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:48.973067 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:48.973068 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:48.973069 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:48.973077 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:48.976182 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:48.976831 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:48.976833 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:48.976833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:48.977310 (+ 477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:48.977520 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:48.977521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:48.977522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:48.977650 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:48.977832 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:48.977868 (+ 36us) server_negotiation.cc:299] Negotiation successful
0619 16:53:48.977954 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":106,"threads_started":1}
I20260619 16:53:48.978168 22643 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jun 19 16:53:48 dist-test-slave-jsw9 krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 16:53:48.980423 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:53:48.981586 22685 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "25f2c9fcad824daa897e9095362eaff0"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41962
I20260619 16:53:48.981751 22685 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25f2c9fcad824daa897e9095362eaff0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 16:53:48.981812 22685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25f2c9fcad824daa897e9095362eaff0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:48.982769 22863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.973350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45877 (local address 127.0.0.1:37194)
0619 16:53:48.973444 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:48.973454 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:48.973492 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:48.973782 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:48.973783 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:48.973938 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:48.973985 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:48.973988 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.975085 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.975086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:48.975891 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:48.975894 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.976422 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.976423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:48.976473 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:48.976830 (+ 357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:53:48.976836 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:48.979351 (+ 2515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:48.981092 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:48.981093 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:48.981093 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:48.981250 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:48.981406 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:48.981407 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:48.981407 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:48.981462 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:48.982447 (+ 985us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:48.982448 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:48.982488 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 16:53:48.982547 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 16:53:48.982655 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 16:53:48.982817 22866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.973465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37194 (local address 127.0.98.60:45877)
0619 16:53:48.973650 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:48.973652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:48.973660 (+ 8us) server_negotiation.cc:407] Connection header received
0619 16:53:48.973681 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:48.973682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:48.973694 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:48.973741 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:48.974069 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.974971 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.975985 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.976317 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.976374 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:48.979600 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:48.979601 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:48.979601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:48.979609 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:48.981017 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:48.981307 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:48.981308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:48.981309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:48.981357 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:48.982255 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:48.982256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:48.982257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:48.982384 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:48.982449 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:48.982573 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 16:53:48.982716 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":163}
I20260619 16:53:48.983296 22685 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25f2c9fcad824daa897e9095362eaff0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25f2c9fcad824daa897e9095362eaff0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46517 } } peers { permanent_uuid: "6d0102ef76554921ad64934924015fc1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } } peers { permanent_uuid: "ba6efe458efb4c488d47cd11e8f782bc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45877 } }
I20260619 16:53:48.983608 22685 leader_election.cc:288] T 00000000000000000000000000000000 P 25f2c9fcad824daa897e9095362eaff0 [CANDIDATE]: Term 1 election: Requested vote from peers 6d0102ef76554921ad64934924015fc1 (127.0.98.61:43123), ba6efe458efb4c488d47cd11e8f782bc (127.0.98.60:45877)
I20260619 16:53:48.984586 22861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba6efe458efb4c488d47cd11e8f782bc: Bootstrap starting.
I20260619 16:53:48.985499 22861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba6efe458efb4c488d47cd11e8f782bc: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:48.985949 22861 log.cc:824] T 00000000000000000000000000000000 P ba6efe458efb4c488d47cd11e8f782bc: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:48.986924 22861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba6efe458efb4c488d47cd11e8f782bc: No bootstrap required, opened a new log
I20260619 16:53:48.989328 22861 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba6efe458efb4c488d47cd11e8f782bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25f2c9fcad824daa897e9095362eaff0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46517 } } peers { permanent_uuid: "6d0102ef76554921ad64934924015fc1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } } peers { permanent_uuid: "ba6efe458efb4c488d47cd11e8f782bc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45877 } }
I20260619 16:53:48.989516 22861 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba6efe458efb4c488d47cd11e8f782bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba6efe458efb4c488d47cd11e8f782bc, State: Initialized, Role: FOLLOWER
I20260619 16:53:48.989615 22861 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba6efe458efb4c488d47cd11e8f782bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25f2c9fcad824daa897e9095362eaff0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46517 } } peers { permanent_uuid: "6d0102ef76554921ad64934924015fc1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } } peers { permanent_uuid: "ba6efe458efb4c488d47cd11e8f782bc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45877 } }
I20260619 16:53:48.990082 22861 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba6efe458efb4c488d47cd11e8f782bc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:53:48.990144 22835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "25f2c9fcad824daa897e9095362eaff0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ba6efe458efb4c488d47cd11e8f782bc"
I20260619 16:53:48.990278 22835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba6efe458efb4c488d47cd11e8f782bc [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:48.991334 22835 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ba6efe458efb4c488d47cd11e8f782bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25f2c9fcad824daa897e9095362eaff0 in term 1.
I20260619 16:53:48.991676 22660 leader_election.cc:302] T 00000000000000000000000000000000 P 25f2c9fcad824daa897e9095362eaff0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25f2c9fcad824daa897e9095362eaff0, ba6efe458efb4c488d47cd11e8f782bc; no voters:
I20260619 16:53:48.991869 22869 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25f2c9fcad824daa897e9095362eaff0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:53:48.992057 22869 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25f2c9fcad824daa897e9095362eaff0 [term 1 LEADER]: Becoming Leader. State: Replica: 25f2c9fcad824daa897e9095362eaff0, State: Running, Role: LEADER
I20260619 16:53:48.992179 22869 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25f2c9fcad824daa897e9095362eaff0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25f2c9fcad824daa897e9095362eaff0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46517 } } peers { permanent_uuid: "6d0102ef76554921ad64934924015fc1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } } peers { permanent_uuid: "ba6efe458efb4c488d47cd11e8f782bc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45877 } }
I20260619 16:53:48.992911 22869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25f2c9fcad824daa897e9095362eaff0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25f2c9fcad824daa897e9095362eaff0. Latest consensus state: current_term: 1 leader_uuid: "25f2c9fcad824daa897e9095362eaff0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25f2c9fcad824daa897e9095362eaff0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46517 } } peers { permanent_uuid: "6d0102ef76554921ad64934924015fc1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } } peers { permanent_uuid: "ba6efe458efb4c488d47cd11e8f782bc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45877 } } }
I20260619 16:53:48.993287 22869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25f2c9fcad824daa897e9095362eaff0 [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:48.995312 22890 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba6efe458efb4c488d47cd11e8f782bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25f2c9fcad824daa897e9095362eaff0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46517 } } peers { permanent_uuid: "6d0102ef76554921ad64934924015fc1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } } peers { permanent_uuid: "ba6efe458efb4c488d47cd11e8f782bc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45877 } } }
I20260619 16:53:48.995442 22890 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba6efe458efb4c488d47cd11e8f782bc [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:53:48.995733 22904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:53:48.996281 22904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260619 16:53:48.996546 22901 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ba6efe458efb4c488d47cd11e8f782bc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:53:48.996610 22901 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:53:48.998592 22835 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ba6efe458efb4c488d47cd11e8f782bc [term 1 FOLLOWER]: Refusing update from remote peer 25f2c9fcad824daa897e9095362eaff0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:53:48.999028 22869 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 25f2c9fcad824daa897e9095362eaff0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba6efe458efb4c488d47cd11e8f782bc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:53:48.997383 22793 master_runner.cc:186] Error getting master registration for 127.0.98.61:43123: OK, instance_id { permanent_uuid: "6d0102ef76554921ad64934924015fc1" instance_seqno: 1781888028774325 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 16:53:49.000054 22793 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 16:53:49.001384 22890 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba6efe458efb4c488d47cd11e8f782bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25f2c9fcad824daa897e9095362eaff0. Latest consensus state: current_term: 1 leader_uuid: "25f2c9fcad824daa897e9095362eaff0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25f2c9fcad824daa897e9095362eaff0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46517 } } peers { permanent_uuid: "6d0102ef76554921ad64934924015fc1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } } peers { permanent_uuid: "ba6efe458efb4c488d47cd11e8f782bc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45877 } } }
I20260619 16:53:49.001554 22890 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba6efe458efb4c488d47cd11e8f782bc [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:53:49.002804 22866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.985820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37210 (local address 127.0.98.60:45877)
0619 16:53:48.985858 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:48.985860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:48.986866 (+ 1006us) server_negotiation.cc:407] Connection header received
0619 16:53:48.986954 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:48.986955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:48.986965 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:48.986992 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:48.989585 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.990334 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.994100 (+ 3766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.994590 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.994667 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:48.999517 (+ 4850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:48.999520 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:48.999520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:48.999529 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:49.001338 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:49.001754 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:49.001755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:49.001756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:49.001821 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:49.002079 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:49.002080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:49.002081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:49.002277 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:49.002376 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:49.002600 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 16:53:49.002709 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 16:53:49.003984 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.985349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40226 (local address 127.0.98.61:43123)
0619 16:53:48.985387 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:48.985389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:48.986914 (+ 1525us) server_negotiation.cc:407] Connection header received
0619 16:53:48.987101 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:48.987103 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:48.987120 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:48.987149 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:48.988266 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.989024 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.994374 (+ 5350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.994712 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.994759 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:49.000138 (+ 5379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:49.000140 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:49.000140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:49.000151 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:49.001876 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:49.002816 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:49.002818 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:49.002819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:49.002901 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:49.003303 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:49.003304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:49.003304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:49.003458 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:49.003565 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:49.003786 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 16:53:49.003886 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 16:53:49.005862 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:48.985174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41970 (local address 127.0.98.62:46517)
0619 16:53:48.985234 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:48.985236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:48.985565 (+ 329us) server_negotiation.cc:407] Connection header received
0619 16:53:48.985738 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:48.985740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:48.985756 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:48.985838 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:48.987687 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.988475 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.991349 (+ 2874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:48.991681 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:48.991723 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:49.001812 (+ 10089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:49.001813 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:49.001814 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:49.001823 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:49.003656 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:49.004661 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:49.004662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:49.004663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:49.004733 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:49.005072 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:49.005073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:49.005073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:49.005219 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:49.005299 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:49.005667 (+ 368us) server_negotiation.cc:299] Negotiation successful
0619 16:53:49.005769 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 16:53:49.006280 22869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25f2c9fcad824daa897e9095362eaff0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "25f2c9fcad824daa897e9095362eaff0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25f2c9fcad824daa897e9095362eaff0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46517 } } peers { permanent_uuid: "6d0102ef76554921ad64934924015fc1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } } peers { permanent_uuid: "ba6efe458efb4c488d47cd11e8f782bc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45877 } } }
I20260619 16:53:49.006392 22869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25f2c9fcad824daa897e9095362eaff0 [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:49.006464 22906 mvcc.cc:205] Tried to move back new op lower bound from 7298613366775255040 to 7298613366752956416. Current Snapshot: MvccSnapshot[applied={T|T < 7298613366775255040}]
I20260619 16:53:49.006749 22890 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba6efe458efb4c488d47cd11e8f782bc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "25f2c9fcad824daa897e9095362eaff0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25f2c9fcad824daa897e9095362eaff0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46517 } } peers { permanent_uuid: "6d0102ef76554921ad64934924015fc1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } } peers { permanent_uuid: "ba6efe458efb4c488d47cd11e8f782bc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45877 } } }
I20260619 16:53:49.006884 22890 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba6efe458efb4c488d47cd11e8f782bc [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:53:49.007015 22907 mvcc.cc:205] Tried to move back new op lower bound from 7298613366775255040 to 7298613366752956416. Current Snapshot: MvccSnapshot[applied={T|T < 7298613366775255040}]
I20260619 16:53:49.015759 22904 catalog_manager.cc:1347] Generated new cluster ID: 42782c5051a34d21a0842d2e3fa8ae71
I20260619 16:53:49.015854 22904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:53:49.018734 22786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45877 } attempt: 3
Jun 19 16:53:49 dist-test-slave-jsw9 krb5kdc[22630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 16:53:49.028491 22904 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:53:49.029407 22789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:49.019116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45877 (local address 127.0.0.1:37218)
0619 16:53:49.019223 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:49.019237 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:49.019282 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:49.019536 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:49.019537 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:49.019752 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:49.019825 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:49.019828 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:49.020843 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:49.020845 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:49.021759 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:49.021763 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:49.022304 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:49.022306 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:49.022371 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:49.023758 (+ 1387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:53:49.023764 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:49.025685 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:49.027905 (+ 2220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:49.027906 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:49.027907 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:49.028173 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:49.028503 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:49.028504 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:49.028504 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:49.028577 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:49.028961 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:49.028962 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:49.029036 (+ 74us) client_negotiation.cc:769] Sending connection context
0619 16:53:49.029115 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 16:53:49.029250 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 16:53:49.029418 22866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:49.019356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37218 (local address 127.0.98.60:45877)
0619 16:53:49.019402 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:49.019404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:49.019413 (+ 9us) server_negotiation.cc:407] Connection header received
0619 16:53:49.019439 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:49.019440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:49.019451 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:49.019899 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:49.019942 (+ 43us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:49.020734 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:49.021883 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:49.022218 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:49.022606 (+ 388us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:49.025797 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:49.025799 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:49.025799 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:49.025808 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:49.027786 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:49.028272 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:49.028296 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:49.028297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:49.028389 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:49.028728 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:49.028729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:49.028729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:49.028893 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:49.028957 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:49.029163 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 16:53:49.029289 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 16:53:49.029403 22904 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:53:49.031849 22786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d0102ef76554921ad64934924015fc1: Bootstrap starting.
I20260619 16:53:49.032775 22786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d0102ef76554921ad64934924015fc1: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:49.033115 22786 log.cc:824] T 00000000000000000000000000000000 P 6d0102ef76554921ad64934924015fc1: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:49.034297 22786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d0102ef76554921ad64934924015fc1: No bootstrap required, opened a new log
I20260619 16:53:49.036531 22786 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d0102ef76554921ad64934924015fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25f2c9fcad824daa897e9095362eaff0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46517 } } peers { permanent_uuid: "6d0102ef76554921ad64934924015fc1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } } peers { permanent_uuid: "ba6efe458efb4c488d47cd11e8f782bc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45877 } }
I20260619 16:53:49.036705 22786 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d0102ef76554921ad64934924015fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d0102ef76554921ad64934924015fc1, State: Initialized, Role: FOLLOWER
I20260619 16:53:49.036819 22786 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d0102ef76554921ad64934924015fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25f2c9fcad824daa897e9095362eaff0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46517 } } peers { permanent_uuid: "6d0102ef76554921ad64934924015fc1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } } peers { permanent_uuid: "ba6efe458efb4c488d47cd11e8f782bc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45877 } }
I20260619 16:53:49.037220 22786 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d0102ef76554921ad64934924015fc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:53:49.037276 22760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "25f2c9fcad824daa897e9095362eaff0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6d0102ef76554921ad64934924015fc1"
I20260619 16:53:49.037381 22760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d0102ef76554921ad64934924015fc1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:49.038938 22760 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6d0102ef76554921ad64934924015fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25f2c9fcad824daa897e9095362eaff0 in term 1.
I20260619 16:53:49.039039 22913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d0102ef76554921ad64934924015fc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25f2c9fcad824daa897e9095362eaff0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46517 } } peers { permanent_uuid: "6d0102ef76554921ad64934924015fc1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } } peers { permanent_uuid: "ba6efe458efb4c488d47cd11e8f782bc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45877 } } }
I20260619 16:53:49.039115 22913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d0102ef76554921ad64934924015fc1 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:53:49.039736 22759 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6d0102ef76554921ad64934924015fc1 [term 1 FOLLOWER]: Refusing update from remote peer 25f2c9fcad824daa897e9095362eaff0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:53:49.040184 22869 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 25f2c9fcad824daa897e9095362eaff0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d0102ef76554921ad64934924015fc1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:53:49.042651 22913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d0102ef76554921ad64934924015fc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25f2c9fcad824daa897e9095362eaff0. Latest consensus state: current_term: 1 leader_uuid: "25f2c9fcad824daa897e9095362eaff0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25f2c9fcad824daa897e9095362eaff0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46517 } } peers { permanent_uuid: "6d0102ef76554921ad64934924015fc1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } } peers { permanent_uuid: "ba6efe458efb4c488d47cd11e8f782bc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45877 } } }
I20260619 16:53:49.042750 22913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d0102ef76554921ad64934924015fc1 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:53:49.042783 22913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d0102ef76554921ad64934924015fc1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "25f2c9fcad824daa897e9095362eaff0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25f2c9fcad824daa897e9095362eaff0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46517 } } peers { permanent_uuid: "6d0102ef76554921ad64934924015fc1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } } peers { permanent_uuid: "ba6efe458efb4c488d47cd11e8f782bc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45877 } } }
I20260619 16:53:49.042817 22913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d0102ef76554921ad64934924015fc1 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:53:49.043296 22924 catalog_manager.cc:1259] Loaded cluster ID: 42782c5051a34d21a0842d2e3fa8ae71
I20260619 16:53:49.043331 22924 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6d0102ef76554921ad64934924015fc1: loading cluster ID for follower catalog manager: success
I20260619 16:53:49.044174 22924 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6d0102ef76554921ad64934924015fc1: acquiring CA information for follower catalog manager: success
W20260619 16:53:49.044263 22924 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260619 16:53:49.069881 22869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25f2c9fcad824daa897e9095362eaff0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "25f2c9fcad824daa897e9095362eaff0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25f2c9fcad824daa897e9095362eaff0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46517 } } peers { permanent_uuid: "6d0102ef76554921ad64934924015fc1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43123 } } peers { permanent_uuid: "ba6efe458efb4c488d47cd11e8f782bc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45877 } } }
I20260619 16:53:49.070039 22869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25f2c9fcad824daa897e9095362eaff0 [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:49.072480 22904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25f2c9fcad824daa897e9095362eaff0: Generated new TSK 0
I20260619 16:53:49.072796 22904 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 16:53:49.101688 22714 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:53:49.099475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41982 (local address 127.0.98.62:46517)
0619 16:53:49.099533 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:49.099535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:49.100818 (+ 1283us) server_negotiation.cc:407] Connection header received
0619 16:53:49.100934 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:49.100935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:49.100950 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:49.100982 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:49.101531 (+ 549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41982: BlockingRecv error: recv got EOF from 127.0.0.1:41982 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260619 16:53:49.102938 22868 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:53:49.100044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40230 (local address 127.0.98.61:43123)
0619 16:53:49.100096 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:49.100099 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:49.102176 (+ 2077us) server_negotiation.cc:407] Connection header received
0619 16:53:49.102328 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:49.102330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:49.102343 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:49.102375 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:49.102855 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40230: BlockingRecv error: recv got EOF from 127.0.0.1:40230 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260619 16:53:49.103955 22866 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:53:49.100479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37220 (local address 127.0.98.60:45877)
0619 16:53:49.100527 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:49.100529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:49.103376 (+ 2847us) server_negotiation.cc:407] Connection header received
0619 16:53:49.103475 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:49.103477 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:49.103490 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:49.103521 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:49.103882 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37220: BlockingRecv error: recv got EOF from 127.0.0.1:37220 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260619 16:53:49.107506 22868 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:53:49.106520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40246 (local address 127.0.98.61:43123)
0619 16:53:49.106573 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:49.106576 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:49.106806 (+ 230us) server_negotiation.cc:407] Connection header received
0619 16:53:49.106924 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:49.106925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:49.106950 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:49.106981 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:49.107429 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40246: BlockingRecv error: recv got EOF from 127.0.0.1:40246 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260619 16:53:49.111182 22868 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:53:49.110144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40252 (local address 127.0.98.61:43123)
0619 16:53:49.110224 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:49.110226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:49.110547 (+ 321us) server_negotiation.cc:407] Connection header received
0619 16:53:49.110665 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:49.110666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:49.110679 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:49.110709 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:49.111114 (+ 405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40252: BlockingRecv error: recv got EOF from 127.0.0.1:40252 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260619 16:53:49.115610 22868 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:53:49.114692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40256 (local address 127.0.98.61:43123)
0619 16:53:49.114734 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:49.114736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:49.115022 (+ 286us) server_negotiation.cc:407] Connection header received
0619 16:53:49.115122 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:49.115123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:49.115136 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:49.115165 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:49.115529 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40256: BlockingRecv error: recv got EOF from 127.0.0.1:40256 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260619 16:53:49.122062 22868 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:53:49.121102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40258 (local address 127.0.98.61:43123)
0619 16:53:49.121151 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:49.121154 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:49.121456 (+ 302us) server_negotiation.cc:407] Connection header received
0619 16:53:49.121558 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:49.121559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:49.121585 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:49.121614 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:49.121996 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40258: BlockingRecv error: recv got EOF from 127.0.0.1:40258 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260619 16:53:49.132774 22868 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:53:49.131670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40268 (local address 127.0.98.61:43123)
0619 16:53:49.131720 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:49.131722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:49.132046 (+ 324us) server_negotiation.cc:407] Connection header received
0619 16:53:49.132164 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:49.132166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:49.132181 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:49.132216 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:49.132694 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40268: BlockingRecv error: recv got EOF from 127.0.0.1:40268 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260619 16:53:49.151827 22868 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:53:49.150712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40274 (local address 127.0.98.61:43123)
0619 16:53:49.150773 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:49.150775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:49.151121 (+ 346us) server_negotiation.cc:407] Connection header received
0619 16:53:49.151231 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:49.151232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:49.151247 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:49.151277 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:49.151751 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40274: BlockingRecv error: recv got EOF from 127.0.0.1:40274 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260619 16:53:49.186568 22868 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:53:49.185616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40284 (local address 127.0.98.61:43123)
0619 16:53:49.185672 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:49.185674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:49.185826 (+ 152us) server_negotiation.cc:407] Connection header received
0619 16:53:49.185895 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:49.185896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:49.185910 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:49.185974 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:49.186458 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40284: BlockingRecv error: recv got EOF from 127.0.0.1:40284 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260619 16:53:49.252625 22868 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:53:49.251796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40296 (local address 127.0.98.61:43123)
0619 16:53:49.251884 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:49.251887 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:49.251914 (+ 27us) server_negotiation.cc:407] Connection header received
0619 16:53:49.252007 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:49.252008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:49.252020 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:49.252087 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:49.252548 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40296: BlockingRecv error: recv got EOF from 127.0.0.1:40296 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 16:53:49.382716 22868 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:53:49.381925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40308 (local address 127.0.98.61:43123)
0619 16:53:49.381983 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:49.381985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:49.382088 (+ 103us) server_negotiation.cc:407] Connection header received
0619 16:53:49.382147 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:49.382148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:49.382160 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:49.382258 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:49.382637 (+ 379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40308: BlockingRecv error: recv got EOF from 127.0.0.1:40308 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260619 16:53:49.640937 22868 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:53:49.640236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40322 (local address 127.0.98.61:43123)
0619 16:53:49.640308 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:49.640310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:49.640325 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:53:49.640407 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:49.640408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:49.640420 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:49.640471 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:49.640855 (+ 384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40322: BlockingRecv error: recv got EOF from 127.0.0.1:40322 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 16:53:49.997221 22901 catalog_manager.cc:1259] Loaded cluster ID: 42782c5051a34d21a0842d2e3fa8ae71
I20260619 16:53:49.997355 22901 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ba6efe458efb4c488d47cd11e8f782bc: loading cluster ID for follower catalog manager: success
I20260619 16:53:49.998594 22901 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ba6efe458efb4c488d47cd11e8f782bc: acquiring CA information for follower catalog manager: success
I20260619 16:53:49.999064 22901 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ba6efe458efb4c488d47cd11e8f782bc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:53:50.045104 22924 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6d0102ef76554921ad64934924015fc1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:53:50.157331 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:50.154256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40326 (local address 127.0.98.61:43123)
0619 16:53:50.154393 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:50.154395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:50.154454 (+ 59us) server_negotiation.cc:407] Connection header received
0619 16:53:50.154531 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:50.154532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:50.154605 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:50.154679 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:53:50.154851 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:50.155461 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:50.156335 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:50.156562 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:50.156623 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:50.156860 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:53:50.157057 (+ 197us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:53:50.157147 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:50.157199 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 16:53:50.157228 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
I20260619 16:53:50.162554 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:50.159766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37228 (local address 127.0.98.60:45877)
0619 16:53:50.159893 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:50.159895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:50.159903 (+ 8us) server_negotiation.cc:407] Connection header received
0619 16:53:50.159947 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:50.159948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:50.159994 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:50.160082 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:53:50.160210 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:50.160746 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:50.161518 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:50.161726 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:50.161783 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:50.162113 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:53:50.162296 (+ 183us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:53:50.162382 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:50.162405 (+ 23us) server_negotiation.cc:299] Negotiation successful
0619 16:53:50.162434 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":57,"threads_started":1}
I20260619 16:53:50.168761 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:50.165117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40332 (local address 127.0.98.61:43123)
0619 16:53:50.165482 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:50.165485 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:50.165495 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:53:50.165533 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:50.165534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:50.165549 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:50.165651 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:53:50.166139 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:50.166980 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:50.167978 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:50.168258 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:50.168323 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:50.168452 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:53:50.168487 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:53:50.168653 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:50.168668 (+ 15us) server_negotiation.cc:299] Negotiation successful
0619 16:53:50.168695 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334}
I20260619 16:53:50.168810 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:50.165063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41994 (local address 127.0.98.62:46517)
0619 16:53:50.165239 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:50.165242 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:50.165259 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:53:50.165326 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:50.165327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:50.165406 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:50.165555 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:53:50.165710 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:50.166539 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:50.167663 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:50.167939 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:50.168023 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:50.168249 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:53:50.168390 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:53:50.168448 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:50.168545 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 16:53:50.168576 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":65,"threads_started":1}
I20260619 16:53:50.169607 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:50.165140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37230 (local address 127.0.98.60:45877)
0619 16:53:50.165227 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:50.165230 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:50.165576 (+ 346us) server_negotiation.cc:407] Connection header received
0619 16:53:50.165707 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:50.165709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:50.165723 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:50.165898 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:53:50.166234 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:50.166978 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:50.168836 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:50.169132 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:50.169198 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:50.169340 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:53:50.169380 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:53:50.169428 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:50.169495 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 16:53:50.169525 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 16:53:50.171120 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 22643
I20260619 16:53:50.176719 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 22718
I20260619 16:53:50.182499 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 22793
2026-06-19T16:53:50Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2822 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:53:50 dist-test-slave-jsw9 krb5kdc[23012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:53:50 dist-test-slave-jsw9 krb5kdc[23012](info): set up 2 sockets
Jun 19 16:53:50 dist-test-slave-jsw9 krb5kdc[23012](info): commencing operation
krb5kdc: starting...
W20260619 16:53:51.158588 392 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/krb5kdc/test-user.keytab.
Jun 19 16:53:51 dist-test-slave-jsw9 krb5kdc[23012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:53:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:53:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:53:51.292920 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43359
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43359,127.0.98.61:33413,127.0.98.60:39041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:53:51.384634 23025 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 16:53:51.384922 23025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:51.384940 23025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:51.384964 23025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:51.386375 23025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:53:51.386446 23025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:51.386461 23025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:51.386472 23025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:53:51.386483 23025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:53:51.388603 23025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43359,127.0.98.61:33413,127.0.98.60:39041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:51.388980 23025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:51.389271 23025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:51.395006 23030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:51.394960 23031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:51.395021 23033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:51.395296 23025 server_base.cc:1034] running on GCE node
I20260619 16:53:51.395601 23025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:51.395962 23025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:51.397150 23025 hybrid_clock.cc:648] HybridClock initialized: now 1781888031397126 us; error 40 us; skew 500 ppm
Jun 19 16:53:51 dist-test-slave-jsw9 krb5kdc[23012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:51.403659 23025 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:53:51.404233 23025 webserver.cc:460] Webserver started at http://127.0.98.62:43225/ using document root <none> and password file <none>
I20260619 16:53:51.404505 23025 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:51.404560 23025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:51.404698 23025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:51.405987 23025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-0/data/instance:
uuid: "94f5d38aadaa4469ae7cfe8bedf10346"
format_stamp: "Formatted at 2026-06-19 16:53:51 on dist-test-slave-jsw9"
server_key: "e3cd20f3e476fe9d02d229222249b105"
server_key_iv: "c931685435b1d9d59eeeb77dcd385604"
server_key_version: "encryptionkey@0"
I20260619 16:53:51.406394 23025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-0/wal/instance:
uuid: "94f5d38aadaa4469ae7cfe8bedf10346"
format_stamp: "Formatted at 2026-06-19 16:53:51 on dist-test-slave-jsw9"
server_key: "e3cd20f3e476fe9d02d229222249b105"
server_key_iv: "c931685435b1d9d59eeeb77dcd385604"
server_key_version: "encryptionkey@0"
I20260619 16:53:51.407850 23025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:53:51.408679 23040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:51.408864 23025 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:53:51.408968 23025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-0/wal
uuid: "94f5d38aadaa4469ae7cfe8bedf10346"
format_stamp: "Formatted at 2026-06-19 16:53:51 on dist-test-slave-jsw9"
server_key: "e3cd20f3e476fe9d02d229222249b105"
server_key_iv: "c931685435b1d9d59eeeb77dcd385604"
server_key_version: "encryptionkey@0"
I20260619 16:53:51.409051 23025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:51.421057 23025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:51.423840 23025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:51.423977 23025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:51.430100 23025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43359
I20260619 16:53:51.430137 23092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43359 every 8 connection(s)
I20260619 16:53:51.430640 23025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:53:51.431842 23093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:51.433773 23093 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43359 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:53:51.439322 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 23025
I20260619 16:53:51.439433 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-0/wal/instance
I20260619 16:53:51.439633 392 external_mini_cluster.cc:1428] Setting key c9e70ad9ce5cd4b728f8030808639b2f
Jun 19 16:53:51 dist-test-slave-jsw9 krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:53:51.445678 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.434700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57432 (local address 127.0.98.62:43359)
0619 16:53:51.434887 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:51.434893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:51.435023 (+ 130us) server_negotiation.cc:407] Connection header received
0619 16:53:51.435603 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:51.435606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:51.435626 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:51.435712 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:51.436363 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.437321 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.438344 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.438695 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.438798 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:51.441393 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:51.441402 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:51.441402 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:51.441409 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:51.443497 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:51.444230 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:51.444233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:51.444234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:51.444349 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:51.444642 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:51.444643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:51.444643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:51.444955 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:51.445031 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:51.445286 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 16:53:51.445422 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":89,"threads_started":1}
I20260619 16:53:51.445691 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.434612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43359 (local address 127.0.0.1:57432)
0619 16:53:51.434824 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:51.434873 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:51.435443 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:51.435723 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:51.435725 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:51.436025 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:51.436236 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:51.436241 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.437421 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.437426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:51.438269 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:51.438272 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.438799 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.438800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:51.438859 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:51.439594 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:53:51.439606 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:51.441301 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:51.443612 (+ 2311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:51.443622 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:51.443625 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:51.443918 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:51.444431 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:51.444432 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:51.444432 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:51.444563 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:51.445027 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:51.445034 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:51.445135 (+ 101us) client_negotiation.cc:769] Sending connection context
0619 16:53:51.445264 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 16:53:51.445503 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":49,"thread_start_us":99,"threads_started":1}
I20260619 16:53:51.446604 23093 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33413 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:53:51.447018 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.446884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33413 (local address 127.0.0.1:56508)
0619 16:53:51.446934 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:51.446988 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260619 16:53:51.447202 23044 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33413: connect: Connection refused (error 111)
I20260619 16:53:51.447700 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.447593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33413 (local address 127.0.0.1:56522)
0619 16:53:51.447642 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:51.447676 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260619 16:53:51.447858 23093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33413: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33413: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 16:53:51.469889 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33413
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43359,127.0.98.61:33413,127.0.98.60:39041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:53:51.497144 23093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33413 } attempt: 1
I20260619 16:53:51.497838 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.497641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33413 (local address 127.0.0.1:56530)
0619 16:53:51.497725 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:51.497789 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 16:53:51.498349 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.498239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33413 (local address 127.0.0.1:56540)
0619 16:53:51.498270 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:51.498324 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260619 16:53:51.498458 23093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33413: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33413: connect: Connection refused (error 111)
W20260619 16:53:51.566126 23100 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 16:53:51.566476 23100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:51.566499 23100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:51.566538 23100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 16:53:51.566628 23093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33413 } attempt: 2
I20260619 16:53:51.567157 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.566990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33413 (local address 127.0.0.1:56554)
0619 16:53:51.567057 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:51.567112 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 16:53:51.567535 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.567440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33413 (local address 127.0.0.1:56566)
0619 16:53:51.567484 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:51.567513 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 16:53:51.567691 23093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33413: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33413: connect: Connection refused (error 111)
W20260619 16:53:51.568249 23100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:53:51.568293 23100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:51.568307 23100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:51.568320 23100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:53:51.568331 23100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:53:51.570094 23100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43359,127.0.98.61:33413,127.0.98.60:39041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:51.570413 23100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:51.570674 23100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:51.576573 23105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:51.576572 23106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:51.576684 23100 server_base.cc:1034] running on GCE node
W20260619 16:53:51.576781 23108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:51.577145 23100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:51.577510 23100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:51.578712 23100 hybrid_clock.cc:648] HybridClock initialized: now 1781888031578699 us; error 31 us; skew 500 ppm
Jun 19 16:53:51 dist-test-slave-jsw9 krb5kdc[23012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:51.581766 23100 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 16:53:51.582386 23100 webserver.cc:460] Webserver started at http://127.0.98.61:44347/ using document root <none> and password file <none>
I20260619 16:53:51.582640 23100 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:51.582690 23100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:51.582820 23100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:51.584017 23100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-1/data/instance:
uuid: "92754a93811c4c748368ceaa258d80f6"
format_stamp: "Formatted at 2026-06-19 16:53:51 on dist-test-slave-jsw9"
server_key: "550e5f0872ba24c317c315b9018d2231"
server_key_iv: "38c4c23e62b7d411e3d21d6d3cecb2a9"
server_key_version: "encryptionkey@0"
I20260619 16:53:51.584403 23100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-1/wal/instance:
uuid: "92754a93811c4c748368ceaa258d80f6"
format_stamp: "Formatted at 2026-06-19 16:53:51 on dist-test-slave-jsw9"
server_key: "550e5f0872ba24c317c315b9018d2231"
server_key_iv: "38c4c23e62b7d411e3d21d6d3cecb2a9"
server_key_version: "encryptionkey@0"
I20260619 16:53:51.585844 23100 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 16:53:51.586694 23115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:51.586861 23100 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 16:53:51.586952 23100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-1/wal
uuid: "92754a93811c4c748368ceaa258d80f6"
format_stamp: "Formatted at 2026-06-19 16:53:51 on dist-test-slave-jsw9"
server_key: "550e5f0872ba24c317c315b9018d2231"
server_key_iv: "38c4c23e62b7d411e3d21d6d3cecb2a9"
server_key_version: "encryptionkey@0"
I20260619 16:53:51.587018 23100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:51.605772 23100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:51.608304 23100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:51.608467 23100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:51.613297 23100 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33413
I20260619 16:53:51.613505 23167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33413 every 8 connection(s)
I20260619 16:53:51.613799 23100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-1/data/info.pb
I20260619 16:53:51.614984 23168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:51.616721 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 23100
I20260619 16:53:51.616832 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-1/wal/instance
I20260619 16:53:51.617038 392 external_mini_cluster.cc:1428] Setting key 7f24752258900ee93de93f932ba7081b
I20260619 16:53:51.616852 23168 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43359 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:53:51 dist-test-slave-jsw9 krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260619 16:53:51.628212 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.617760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57442 (local address 127.0.98.62:43359)
0619 16:53:51.617822 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:51.617825 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:51.618087 (+ 262us) server_negotiation.cc:407] Connection header received
0619 16:53:51.618806 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:51.618807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:51.618826 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:51.618880 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:51.619915 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.620668 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.621756 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.622130 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.622242 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:51.624879 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:51.624881 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:51.624881 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:51.624893 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:51.626512 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:51.627046 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:51.627047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:51.627048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:51.627116 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:51.627457 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:51.627458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:51.627464 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:51.627611 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:51.627680 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:51.628024 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 16:53:51.628126 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260619 16:53:51.628464 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.617716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43359 (local address 127.0.0.1:57442)
0619 16:53:51.617948 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:51.617993 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:51.618654 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:51.619000 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:51.619005 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:51.619416 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:51.619782 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:51.619789 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.620770 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.620771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:51.621670 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:51.621672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.622286 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.622287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:51.622412 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:51.623010 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:53:51.623020 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:51.624764 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:51.626637 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:51.626648 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:51.626655 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:51.626934 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:51.627210 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:51.627211 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:51.627212 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:51.627345 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:51.627708 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:51.627715 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:51.627879 (+ 164us) client_negotiation.cc:769] Sending connection context
0619 16:53:51.627980 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 16:53:51.628191 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":130,"threads_started":1}
I20260619 16:53:51.629163 23168 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33413 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:53:51 dist-test-slave-jsw9 krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 16:53:51.638067 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.629501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33413 (local address 127.0.0.1:56578)
0619 16:53:51.629550 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:51.629563 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:51.629609 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:51.629884 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:51.629885 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:51.630050 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:51.630121 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:51.630124 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.631733 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.631734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:51.632288 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:51.632289 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.632927 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.632928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:51.632987 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:51.633522 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:53:51.633526 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:51.634907 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:51.636927 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:51.636928 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:51.636928 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:51.637109 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:51.637342 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:51.637343 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:51.637343 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:51.637391 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:51.637673 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:51.637674 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:51.637716 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 16:53:51.637772 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 16:53:51.637900 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 16:53:51.638098 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.629581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56578 (local address 127.0.98.61:33413)
0619 16:53:51.629739 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:51.629744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:51.629759 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:53:51.629776 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:51.629777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:51.629789 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:51.629852 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:51.630470 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.631624 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.632369 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.632851 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.632926 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:51.634989 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:51.635002 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:51.635002 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:51.635009 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:51.636781 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:51.637210 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:51.637228 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:51.637228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:51.637282 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:51.637443 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:51.637443 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:51.637444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:51.637604 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:51.637685 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:51.637833 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 16:53:51.637987 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":134,"thread_start_us":99,"threads_started":1}
I20260619 16:53:51.638887 23168 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39041 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:53:51.639271 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.639154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39041 (local address 127.0.0.1:33276)
0619 16:53:51.639197 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:51.639247 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260619 16:53:51.639390 23116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39041: connect: Connection refused (error 111)
I20260619 16:53:51.639853 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.639757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39041 (local address 127.0.0.1:33286)
0619 16:53:51.639802 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:51.639830 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 16:53:51.640054 23168 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39041: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39041: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 16:53:51.645483 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
/tmp/dist-test-taskjYBH_Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39041
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43359,127.0.98.61:33413,127.0.98.60:39041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:53:51.658885 23093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33413 } attempt: 3
Jun 19 16:53:51 dist-test-slave-jsw9 krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 16:53:51.667140 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.659464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56582 (local address 127.0.98.61:33413)
0619 16:53:51.659536 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:51.659538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:51.659547 (+ 9us) server_negotiation.cc:407] Connection header received
0619 16:53:51.659576 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:51.659577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:51.659587 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:51.659657 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:51.660036 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.660577 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.661326 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.661557 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.661623 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:51.664342 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:51.664344 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:51.664344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:51.664364 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:51.665868 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:51.666303 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:51.666305 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:51.666305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:51.666365 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:51.666608 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:51.666609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:51.666609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:51.666740 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:51.666796 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:51.666960 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 16:53:51.667035 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 16:53:51.667160 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.659357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33413 (local address 127.0.0.1:56582)
0619 16:53:51.659421 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:51.659434 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:51.659481 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:51.659673 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:51.659674 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:51.659867 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:51.659958 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:51.659960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.660674 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.660676 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:51.661243 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:51.661246 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.661627 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.661628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:51.661710 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:51.662285 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:53:51.662292 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:51.664214 (+ 1922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:51.665980 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:51.665982 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:51.665982 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:51.666182 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:51.666444 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:51.666445 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:51.666446 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:51.666532 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:51.666799 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:51.666800 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:51.666868 (+ 68us) client_negotiation.cc:769] Sending connection context
0619 16:53:51.666932 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 16:53:51.667058 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 16:53:51.667649 23093 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39041 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:53:51.668125 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.667947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39041 (local address 127.0.0.1:33290)
0619 16:53:51.668055 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:51.668094 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 16:53:51.668543 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.668404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39041 (local address 127.0.0.1:33304)
0619 16:53:51.668483 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:51.668517 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 16:53:51.668685 23093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39041: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39041: connect: Connection refused (error 111)
I20260619 16:53:51.689369 23168 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39041 } attempt: 1
I20260619 16:53:51.690025 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.689862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39041 (local address 127.0.0.1:33316)
0619 16:53:51.689943 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:51.689986 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 16:53:51.690402 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.690313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39041 (local address 127.0.0.1:33320)
0619 16:53:51.690356 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:51.690383 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 16:53:51.690480 23168 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39041: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39041: connect: Connection refused (error 111)
I20260619 16:53:51.699862 23093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39041 } attempt: 1
I20260619 16:53:51.700557 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.700378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39041 (local address 127.0.0.1:33336)
0619 16:53:51.700455 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:51.700521 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 16:53:51.700934 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.700842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39041 (local address 127.0.0.1:33338)
0619 16:53:51.700884 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:51.700910 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 16:53:51.701032 23093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39041: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39041: connect: Connection refused (error 111)
W20260619 16:53:51.743713 23175 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 16:53:51.743989 23175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:53:51.744014 23175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:53:51.744041 23175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:53:51.745301 23175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:53:51.745353 23175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:53:51.745371 23175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:53:51.745388 23175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:53:51.745402 23175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:53:51.747116 23175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43359,127.0.98.61:33413,127.0.98.60:39041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 15:28:00 UTC on ccycloud.quasar-opyasf.root.comops.site
build id 4455770
I20260619 16:53:51.747437 23175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:53:51.747701 23175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:53:51.753532 23183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:51.753721 23175 server_base.cc:1034] running on GCE node
W20260619 16:53:51.753551 23181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:53:51.753652 23180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:53:51.754140 23175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:53:51.754527 23175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:53:51.755700 23175 hybrid_clock.cc:648] HybridClock initialized: now 1781888031755653 us; error 67 us; skew 500 ppm
Jun 19 16:53:51 dist-test-slave-jsw9 krb5kdc[23012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:53:51.758703 23168 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39041 } attempt: 2
I20260619 16:53:51.759101 23175 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 16:53:51.759192 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.759059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39041 (local address 127.0.0.1:33350)
0619 16:53:51.759121 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:51.759156 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 16:53:51.759744 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.759587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39041 (local address 127.0.0.1:33356)
0619 16:53:51.759655 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:51.759701 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 16:53:51.759768 23175 webserver.cc:460] Webserver started at http://127.0.98.60:45703/ using document root <none> and password file <none>
W20260619 16:53:51.759965 23168 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39041: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39041: connect: Connection refused (error 111)
I20260619 16:53:51.760160 23175 fs_manager.cc:359] Metadata directory not provided
I20260619 16:53:51.760332 23175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:53:51.760473 23175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:53:51.761677 23175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-2/data/instance:
uuid: "9a354dcae9dd4dab8240107c4958a624"
format_stamp: "Formatted at 2026-06-19 16:53:51 on dist-test-slave-jsw9"
server_key: "1670613e2788774a19b69b12b5a8652f"
server_key_iv: "78accb8791254c948374baaa61457e4e"
server_key_version: "encryptionkey@0"
I20260619 16:53:51.761999 23175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-2/wal/instance:
uuid: "9a354dcae9dd4dab8240107c4958a624"
format_stamp: "Formatted at 2026-06-19 16:53:51 on dist-test-slave-jsw9"
server_key: "1670613e2788774a19b69b12b5a8652f"
server_key_iv: "78accb8791254c948374baaa61457e4e"
server_key_version: "encryptionkey@0"
I20260619 16:53:51.763454 23175 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 16:53:51.764328 23190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:53:51.764505 23175 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:53:51.764609 23175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-2/wal
uuid: "9a354dcae9dd4dab8240107c4958a624"
format_stamp: "Formatted at 2026-06-19 16:53:51 on dist-test-slave-jsw9"
server_key: "1670613e2788774a19b69b12b5a8652f"
server_key_iv: "78accb8791254c948374baaa61457e4e"
server_key_version: "encryptionkey@0"
I20260619 16:53:51.764678 23175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:53:51.776201 23093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39041 } attempt: 2
I20260619 16:53:51.776850 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.776691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39041 (local address 127.0.0.1:33360)
0619 16:53:51.776757 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:51.776811 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 16:53:51.777266 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.777184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39041 (local address 127.0.0.1:33368)
0619 16:53:51.777223 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:51.777248 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 16:53:51.777379 23093 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39041: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39041: connect: Connection refused (error 111)
I20260619 16:53:51.782560 23175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:53:51.784865 23175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:53:51.785022 23175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:53:51.789722 23175 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39041
I20260619 16:53:51.789765 23242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39041 every 8 connection(s)
I20260619 16:53:51.790176 23175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-2/data/info.pb
I20260619 16:53:51.791465 23243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:53:51.792534 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu as pid 23175
I20260619 16:53:51.792644 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjYBH_Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781887741903642-392-0/minicluster-data/master-2/wal/instance
I20260619 16:53:51.792834 392 external_mini_cluster.cc:1428] Setting key 3c5a4b140da25d60339cb1389f824f05
I20260619 16:53:51.793620 23243 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43359 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:53:51 dist-test-slave-jsw9 krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 19 16:53:51 dist-test-slave-jsw9 krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 16:53:51.803979 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.794990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57456 (local address 127.0.98.62:43359)
0619 16:53:51.795050 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:51.795053 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:51.795165 (+ 112us) server_negotiation.cc:407] Connection header received
0619 16:53:51.795303 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:51.795305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:51.795318 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:51.795379 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:51.796083 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.796620 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.797610 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.797929 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.798016 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:51.800627 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:51.800629 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:51.800629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:51.800640 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:51.802291 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:51.802875 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:51.802877 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:51.802877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:51.802943 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:51.803223 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:51.803224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:51.803224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:51.803363 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:51.803426 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:51.803774 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 16:53:51.803868 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 16:53:51.804260 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.794776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43359 (local address 127.0.0.1:57456)
0619 16:53:51.795021 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:51.795071 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:51.795221 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:51.795431 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:51.795435 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:51.795767 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:51.795970 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:51.795975 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.796707 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.796708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:51.797509 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:51.797512 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.798078 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.798080 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:51.798245 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:51.798853 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:53:51.798865 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:51.800492 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:51.802368 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:51.802374 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:51.802377 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:51.802762 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:51.803026 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:51.803027 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:51.803027 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:51.803155 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:51.803423 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:51.803431 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:51.803632 (+ 201us) client_negotiation.cc:769] Sending connection context
0619 16:53:51.803721 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 16:53:51.803913 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":3,"thread_start_us":104,"threads_started":1}
I20260619 16:53:51.804293 23246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.793589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33370 (local address 127.0.98.60:39041)
0619 16:53:51.793899 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:51.793912 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:51.794536 (+ 624us) server_negotiation.cc:407] Connection header received
0619 16:53:51.795205 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:51.795214 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:51.795239 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:51.795339 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:51.796046 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.796871 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.797820 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.798175 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.798274 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:51.800981 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:51.800989 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:51.800989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:51.800997 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:51.802854 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:51.803163 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:51.803167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:51.803167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:51.803220 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:51.803379 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:51.803380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:51.803380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:51.803617 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:51.803760 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:51.803845 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 16:53:51.803983 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":79,"server-negotiator.queue_time_us":235,"thread_start_us":100,"threads_started":1}
I20260619 16:53:51.804832 23243 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33413 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:53:51 dist-test-slave-jsw9 krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 19 16:53:51 dist-test-slave-jsw9 krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:53:51.815620 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.805189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56594 (local address 127.0.98.61:33413)
0619 16:53:51.805241 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:51.805243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:51.805357 (+ 114us) server_negotiation.cc:407] Connection header received
0619 16:53:51.805456 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:51.805457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:51.805469 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:51.805498 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:51.805906 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.806612 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.807553 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.807837 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.807907 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:51.810730 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:51.810733 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:51.810733 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:51.810748 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:51.813227 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:51.813968 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:51.813970 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:51.813971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:51.814050 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:51.814446 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:51.814448 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:51.814449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:51.814670 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:51.814843 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:51.815419 (+ 576us) server_negotiation.cc:299] Negotiation successful
0619 16:53:51.815532 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 16:53:51.815713 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.805102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33413 (local address 127.0.0.1:56594)
0619 16:53:51.805294 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:51.805303 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:51.805384 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:51.805562 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:51.805562 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:51.805746 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:51.805808 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:51.805811 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.806717 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.806718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:51.807437 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:51.807441 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.807978 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.807979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:51.808058 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:51.808545 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:53:51.808551 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:51.810458 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:51.813369 (+ 2911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:51.813370 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:51.813371 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:51.813825 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:51.814232 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:51.814234 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:51.814234 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:51.814332 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:51.815210 (+ 878us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:51.815212 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:51.815278 (+ 66us) client_negotiation.cc:769] Sending connection context
0619 16:53:51.815343 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 16:53:51.815601 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177}
I20260619 16:53:51.816192 23243 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39041 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:53:51.816731 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.805147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57470 (local address 127.0.98.62:43359)
0619 16:53:51.805185 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:51.805186 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:51.805215 (+ 29us) server_negotiation.cc:407] Connection header received
0619 16:53:51.805275 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:51.805276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:51.805285 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:51.805335 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:51.805899 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.806672 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.807648 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.807940 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.807983 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:51.812246 (+ 4263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:51.812250 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:51.812250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:51.812262 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:51.814672 (+ 2410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:51.815205 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:51.815206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:51.815207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:51.815285 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:51.815727 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:51.815728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:51.815729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:51.816016 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:51.816460 (+ 444us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:51.816536 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 16:53:51.816628 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Jun 19 16:53:51 dist-test-slave-jsw9 krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 16:53:51 dist-test-slave-jsw9 krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 16:53:51.824580 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.816495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39041 (local address 127.0.0.1:33386)
0619 16:53:51.816655 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:51.816669 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:51.816729 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:51.816905 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:51.816907 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:51.817096 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:51.817159 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:51.817162 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.818294 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.818295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:51.818990 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:51.818993 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.819493 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.819494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:51.819567 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:51.820022 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:53:51.820027 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:51.821697 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:51.823511 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:51.823512 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:51.823512 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:51.823731 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:51.823978 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:51.823979 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:51.823979 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:51.824035 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:51.824296 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:51.824297 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:51.824339 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 16:53:51.824390 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 16:53:51.824491 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260619 16:53:51.824672 23246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.816532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33386 (local address 127.0.98.60:39041)
0619 16:53:51.816578 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:51.816580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:51.816719 (+ 139us) server_negotiation.cc:407] Connection header received
0619 16:53:51.816788 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:51.816789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:51.816800 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:51.816848 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:51.817262 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.818152 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.819072 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.819396 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.819471 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:51.821810 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:51.821812 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:51.821812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:51.821820 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:51.823407 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:51.823816 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:51.823817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:51.823818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:51.823883 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:51.824092 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:51.824093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:51.824094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:51.824236 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:51.824302 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:51.824427 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 16:53:51.824552 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":18}
I20260619 16:53:51.826473 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.817953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56600 (local address 127.0.98.61:33413)
0619 16:53:51.818006 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:51.818008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:51.818033 (+ 25us) server_negotiation.cc:407] Connection header received
0619 16:53:51.818054 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:51.818055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:51.818064 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:51.818107 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:51.818834 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.819515 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.820465 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.820695 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.820768 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:51.823337 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:51.823339 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:51.823340 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:51.823350 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:51.824960 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:51.825332 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:51.825333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:51.825334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:51.825400 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:51.825655 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:51.825656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:51.825657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:51.825817 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:51.826011 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:51.826175 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 16:53:51.826327 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 16:53:51.826862 23243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a354dcae9dd4dab8240107c4958a624: Bootstrap starting.
I20260619 16:53:51.828171 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:53:51.828241 23243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a354dcae9dd4dab8240107c4958a624: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:51.828572 23243 log.cc:824] T 00000000000000000000000000000000 P 9a354dcae9dd4dab8240107c4958a624: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:51.829581 23243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a354dcae9dd4dab8240107c4958a624: No bootstrap required, opened a new log
I20260619 16:53:51.829463 23067 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "94f5d38aadaa4469ae7cfe8bedf10346"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57470
I20260619 16:53:51.832119 23243 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a354dcae9dd4dab8240107c4958a624 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f5d38aadaa4469ae7cfe8bedf10346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43359 } } peers { permanent_uuid: "92754a93811c4c748368ceaa258d80f6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33413 } } peers { permanent_uuid: "9a354dcae9dd4dab8240107c4958a624" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39041 } }
I20260619 16:53:51.832533 23243 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a354dcae9dd4dab8240107c4958a624 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a354dcae9dd4dab8240107c4958a624, State: Initialized, Role: FOLLOWER
I20260619 16:53:51.832716 23243 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a354dcae9dd4dab8240107c4958a624 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f5d38aadaa4469ae7cfe8bedf10346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43359 } } peers { permanent_uuid: "92754a93811c4c748368ceaa258d80f6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33413 } } peers { permanent_uuid: "9a354dcae9dd4dab8240107c4958a624" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39041 } }
I20260619 16:53:51.833089 23249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a354dcae9dd4dab8240107c4958a624 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f5d38aadaa4469ae7cfe8bedf10346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43359 } } peers { permanent_uuid: "92754a93811c4c748368ceaa258d80f6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33413 } } peers { permanent_uuid: "9a354dcae9dd4dab8240107c4958a624" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39041 } } }
I20260619 16:53:51.833170 23243 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a354dcae9dd4dab8240107c4958a624 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:53:51.833230 23249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a354dcae9dd4dab8240107c4958a624 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 16:53:51.835724 23260 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9a354dcae9dd4dab8240107c4958a624: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:53:51.835815 23260 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:53:51.836185 23175 master_runner.cc:186] Error getting master registration for 127.0.98.62:43359: OK, instance_id { permanent_uuid: "94f5d38aadaa4469ae7cfe8bedf10346" instance_seqno: 1781888031428694 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 16:53:51.840265 23175 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 16:53:51.851176 23168 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39041 } attempt: 3
Jun 19 16:53:51 dist-test-slave-jsw9 krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 16:53:51.861009 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.851668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39041 (local address 127.0.0.1:33396)
0619 16:53:51.851754 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:51.851771 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:51.851846 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:51.852008 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:51.852009 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:51.852514 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:51.852646 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:51.852651 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.853755 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.853756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:51.854852 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:51.854857 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.855409 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.855411 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:51.855488 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:51.856115 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:53:51.856121 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:51.857822 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:51.859752 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:51.859754 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:51.859754 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:51.859934 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:51.860144 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:51.860145 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:51.860145 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:51.860207 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:51.860571 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:51.860572 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:51.860626 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 16:53:51.860692 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 16:53:51.860890 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 16:53:51.861008 23246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.851764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33396 (local address 127.0.98.60:39041)
0619 16:53:51.851843 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:51.851845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:51.851859 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:53:51.851917 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:51.851918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:51.851931 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:51.851989 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:51.852772 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.853624 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.854968 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.855289 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.855375 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:51.857927 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:51.857930 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:51.857930 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:51.857940 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:51.859669 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:51.860008 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:51.860009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:51.860010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:51.860073 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:51.860263 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:51.860264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:51.860264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:51.860505 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:51.860577 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:51.860724 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 16:53:51.860887 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 16:53:51.863723 23168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92754a93811c4c748368ceaa258d80f6: Bootstrap starting.
I20260619 16:53:51.864655 23168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92754a93811c4c748368ceaa258d80f6: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:51.865000 23168 log.cc:824] T 00000000000000000000000000000000 P 92754a93811c4c748368ceaa258d80f6: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:51.865870 23168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92754a93811c4c748368ceaa258d80f6: No bootstrap required, opened a new log
I20260619 16:53:51.867631 23168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92754a93811c4c748368ceaa258d80f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f5d38aadaa4469ae7cfe8bedf10346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43359 } } peers { permanent_uuid: "92754a93811c4c748368ceaa258d80f6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33413 } } peers { permanent_uuid: "9a354dcae9dd4dab8240107c4958a624" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39041 } }
I20260619 16:53:51.867794 23168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92754a93811c4c748368ceaa258d80f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92754a93811c4c748368ceaa258d80f6, State: Initialized, Role: FOLLOWER
I20260619 16:53:51.867901 23168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92754a93811c4c748368ceaa258d80f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f5d38aadaa4469ae7cfe8bedf10346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43359 } } peers { permanent_uuid: "92754a93811c4c748368ceaa258d80f6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33413 } } peers { permanent_uuid: "9a354dcae9dd4dab8240107c4958a624" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39041 } }
I20260619 16:53:51.868646 23168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92754a93811c4c748368ceaa258d80f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:53:51.870314 23263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92754a93811c4c748368ceaa258d80f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f5d38aadaa4469ae7cfe8bedf10346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43359 } } peers { permanent_uuid: "92754a93811c4c748368ceaa258d80f6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33413 } } peers { permanent_uuid: "9a354dcae9dd4dab8240107c4958a624" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39041 } } }
I20260619 16:53:51.870615 23263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92754a93811c4c748368ceaa258d80f6 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 16:53:51.870853 23274 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 92754a93811c4c748368ceaa258d80f6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:53:51.870899 23274 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:53:51.871281 23100 master_runner.cc:186] Error getting master registration for 127.0.98.62:43359: OK, instance_id { permanent_uuid: "94f5d38aadaa4469ae7cfe8bedf10346" instance_seqno: 1781888031428694 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 16:53:51.874756 23100 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 16:53:51.876576 23093 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39041 } attempt: 3
Jun 19 16:53:51 dist-test-slave-jsw9 krb5kdc[23012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781888031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 16:53:51.886477 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.877072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39041 (local address 127.0.0.1:33406)
0619 16:53:51.877147 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0619 16:53:51.877166 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:53:51.877239 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:53:51.877486 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:53:51.877487 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:53:51.877728 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:53:51.877800 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:51.877803 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.878853 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.878854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:51.879691 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:53:51.879696 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.880242 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.880244 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:53:51.880306 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:51.880916 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:53:51.880922 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:53:51.882833 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:53:51.885050 (+ 2217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:51.885053 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:51.885053 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:51.885315 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:51.885524 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:53:51.885525 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:53:51.885525 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:53:51.885585 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:53:51.885889 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:53:51.885890 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:53:51.885947 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 16:53:51.886147 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 16:53:51.886347 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 16:53:51.887110 23246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.877237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33406 (local address 127.0.98.60:39041)
0619 16:53:51.877326 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:51.877328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:51.877338 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:53:51.877370 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:51.877371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:51.877383 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:51.877428 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:51.877898 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.878736 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.879793 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.880103 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.880167 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:51.882971 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:51.882974 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:51.882974 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:51.882984 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:51.884903 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:51.885398 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:51.885400 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:51.885400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:51.885461 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:51.885645 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:51.885646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:51.885646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:51.885805 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:51.885864 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:51.886899 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0619 16:53:51.886997 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:53:51.889657 23093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94f5d38aadaa4469ae7cfe8bedf10346: Bootstrap starting.
I20260619 16:53:51.890897 23093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94f5d38aadaa4469ae7cfe8bedf10346: Neither blocks nor log segments found. Creating new log.
I20260619 16:53:51.891352 23093 log.cc:824] T 00000000000000000000000000000000 P 94f5d38aadaa4469ae7cfe8bedf10346: Log is configured to *not* fsync() on all Append() calls
I20260619 16:53:51.893280 23093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94f5d38aadaa4469ae7cfe8bedf10346: No bootstrap required, opened a new log
I20260619 16:53:51.895004 23093 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94f5d38aadaa4469ae7cfe8bedf10346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f5d38aadaa4469ae7cfe8bedf10346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43359 } } peers { permanent_uuid: "92754a93811c4c748368ceaa258d80f6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33413 } } peers { permanent_uuid: "9a354dcae9dd4dab8240107c4958a624" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39041 } }
I20260619 16:53:51.895222 23093 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94f5d38aadaa4469ae7cfe8bedf10346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94f5d38aadaa4469ae7cfe8bedf10346, State: Initialized, Role: FOLLOWER
I20260619 16:53:51.895318 23093 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94f5d38aadaa4469ae7cfe8bedf10346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f5d38aadaa4469ae7cfe8bedf10346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43359 } } peers { permanent_uuid: "92754a93811c4c748368ceaa258d80f6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33413 } } peers { permanent_uuid: "9a354dcae9dd4dab8240107c4958a624" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39041 } }
I20260619 16:53:51.895802 23067 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94f5d38aadaa4469ae7cfe8bedf10346 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 16:53:51.895725 23277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94f5d38aadaa4469ae7cfe8bedf10346 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f5d38aadaa4469ae7cfe8bedf10346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43359 } } peers { permanent_uuid: "92754a93811c4c748368ceaa258d80f6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33413 } } peers { permanent_uuid: "9a354dcae9dd4dab8240107c4958a624" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39041 } } }
I20260619 16:53:51.895924 23067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94f5d38aadaa4469ae7cfe8bedf10346 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:51.896060 23277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94f5d38aadaa4469ae7cfe8bedf10346 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:53:51.897131 23067 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94f5d38aadaa4469ae7cfe8bedf10346 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f5d38aadaa4469ae7cfe8bedf10346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43359 } } peers { permanent_uuid: "92754a93811c4c748368ceaa258d80f6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33413 } } peers { permanent_uuid: "9a354dcae9dd4dab8240107c4958a624" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39041 } }
I20260619 16:53:51.897464 23067 leader_election.cc:288] T 00000000000000000000000000000000 P 94f5d38aadaa4469ae7cfe8bedf10346 [CANDIDATE]: Term 1 election: Requested vote from peers 92754a93811c4c748368ceaa258d80f6 (127.0.98.61:33413), 9a354dcae9dd4dab8240107c4958a624 (127.0.98.60:39041)
I20260619 16:53:51.895754 23093 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94f5d38aadaa4469ae7cfe8bedf10346 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:53:51.898219 23142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "94f5d38aadaa4469ae7cfe8bedf10346" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "92754a93811c4c748368ceaa258d80f6"
I20260619 16:53:51.898386 23142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92754a93811c4c748368ceaa258d80f6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:51.899498 23142 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 92754a93811c4c748368ceaa258d80f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94f5d38aadaa4469ae7cfe8bedf10346 in term 1.
I20260619 16:53:51.899806 23217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "94f5d38aadaa4469ae7cfe8bedf10346" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9a354dcae9dd4dab8240107c4958a624"
I20260619 16:53:51.899933 23217 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a354dcae9dd4dab8240107c4958a624 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:53:51.901367 23217 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9a354dcae9dd4dab8240107c4958a624 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94f5d38aadaa4469ae7cfe8bedf10346 in term 1.
I20260619 16:53:51.902693 23041 leader_election.cc:302] T 00000000000000000000000000000000 P 94f5d38aadaa4469ae7cfe8bedf10346 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94f5d38aadaa4469ae7cfe8bedf10346, 9a354dcae9dd4dab8240107c4958a624; no voters:
I20260619 16:53:51.904335 23277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94f5d38aadaa4469ae7cfe8bedf10346 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:53:51.904924 23025 master_runner.cc:263] No leader master found from master 94f5d38aadaa4469ae7cfe8bedf10346
I20260619 16:53:51.904973 23025 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 16:53:51.905170 23277 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94f5d38aadaa4469ae7cfe8bedf10346 [term 1 LEADER]: Becoming Leader. State: Replica: 94f5d38aadaa4469ae7cfe8bedf10346, State: Running, Role: LEADER
I20260619 16:53:51.905341 23277 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94f5d38aadaa4469ae7cfe8bedf10346 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f5d38aadaa4469ae7cfe8bedf10346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43359 } } peers { permanent_uuid: "92754a93811c4c748368ceaa258d80f6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33413 } } peers { permanent_uuid: "9a354dcae9dd4dab8240107c4958a624" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39041 } }
I20260619 16:53:51.905925 23277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94f5d38aadaa4469ae7cfe8bedf10346 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94f5d38aadaa4469ae7cfe8bedf10346. Latest consensus state: current_term: 1 leader_uuid: "94f5d38aadaa4469ae7cfe8bedf10346" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f5d38aadaa4469ae7cfe8bedf10346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43359 } } peers { permanent_uuid: "92754a93811c4c748368ceaa258d80f6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33413 } } peers { permanent_uuid: "9a354dcae9dd4dab8240107c4958a624" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39041 } } }
I20260619 16:53:51.906069 23277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94f5d38aadaa4469ae7cfe8bedf10346 [sys.catalog]: This master's current role is: LEADER
W20260619 16:53:51.907148 23288 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 94f5d38aadaa4469ae7cfe8bedf10346: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:53:51.907210 23288 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:53:51.907390 23297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:53:51.908151 23297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:53:51.912629 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.903788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57472 (local address 127.0.98.62:43359)
0619 16:53:51.903839 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:51.903842 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:51.904439 (+ 597us) server_negotiation.cc:407] Connection header received
0619 16:53:51.904544 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:51.904546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:51.904557 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:51.904588 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:51.905456 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.906301 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.907346 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.907658 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.907707 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:51.909556 (+ 1849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:51.909557 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:51.909558 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:51.909565 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:51.911261 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:51.911666 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:51.911668 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:51.911668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:51.911727 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:51.911941 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:51.911941 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:51.911942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:51.912100 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:51.912203 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:51.912435 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 16:53:51.912524 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 16:53:51.916476 23246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.904131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33412 (local address 127.0.98.60:39041)
0619 16:53:51.904172 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:51.904174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:51.906397 (+ 2223us) server_negotiation.cc:407] Connection header received
0619 16:53:51.906505 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:51.906506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:51.906517 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:51.906544 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:51.909602 (+ 3058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.910378 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.911479 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.911805 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.911847 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:51.913657 (+ 1810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:51.913658 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:51.913659 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:51.913667 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:51.915280 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:51.915651 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:51.915652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:51.915653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:51.915712 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:51.915884 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:51.915885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:51.915885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:51.916013 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:51.916085 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:51.916274 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 16:53:51.916360 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 16:53:51.918819 23217 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9a354dcae9dd4dab8240107c4958a624 [term 1 FOLLOWER]: Refusing update from remote peer 94f5d38aadaa4469ae7cfe8bedf10346: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:53:51.919266 23277 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 94f5d38aadaa4469ae7cfe8bedf10346 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a354dcae9dd4dab8240107c4958a624" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:53:51.920548 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:51.908467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56612 (local address 127.0.98.61:33413)
0619 16:53:51.908518 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:51.908519 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:51.913446 (+ 4927us) server_negotiation.cc:407] Connection header received
0619 16:53:51.913549 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:51.913551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:51.913564 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:51.913594 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:51.914144 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.914891 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.915883 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:51.916231 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:51.916275 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:51.917640 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:53:51.917641 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:53:51.917642 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:53:51.917649 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:53:51.919267 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:51.919651 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:51.919652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:51.919653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:51.919709 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:53:51.919895 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:53:51.919896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:53:51.919896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:53:51.920034 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:53:51.920140 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:51.920349 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 16:53:51.920443 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 16:53:51.921233 23249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a354dcae9dd4dab8240107c4958a624 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94f5d38aadaa4469ae7cfe8bedf10346. Latest consensus state: current_term: 1 leader_uuid: "94f5d38aadaa4469ae7cfe8bedf10346" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f5d38aadaa4469ae7cfe8bedf10346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43359 } } peers { permanent_uuid: "92754a93811c4c748368ceaa258d80f6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33413 } } peers { permanent_uuid: "9a354dcae9dd4dab8240107c4958a624" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39041 } } }
I20260619 16:53:51.921339 23249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a354dcae9dd4dab8240107c4958a624 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:53:51.921670 23142 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 92754a93811c4c748368ceaa258d80f6 [term 1 FOLLOWER]: Refusing update from remote peer 94f5d38aadaa4469ae7cfe8bedf10346: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:53:51.922048 23295 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 94f5d38aadaa4469ae7cfe8bedf10346 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92754a93811c4c748368ceaa258d80f6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:53:51.922942 23295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94f5d38aadaa4469ae7cfe8bedf10346 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "94f5d38aadaa4469ae7cfe8bedf10346" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f5d38aadaa4469ae7cfe8bedf10346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43359 } } peers { permanent_uuid: "92754a93811c4c748368ceaa258d80f6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33413 } } peers { permanent_uuid: "9a354dcae9dd4dab8240107c4958a624" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39041 } } }
I20260619 16:53:51.923049 23295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94f5d38aadaa4469ae7cfe8bedf10346 [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:51.923516 23249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a354dcae9dd4dab8240107c4958a624 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "94f5d38aadaa4469ae7cfe8bedf10346" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f5d38aadaa4469ae7cfe8bedf10346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43359 } } peers { permanent_uuid: "92754a93811c4c748368ceaa258d80f6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33413 } } peers { permanent_uuid: "9a354dcae9dd4dab8240107c4958a624" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39041 } } }
I20260619 16:53:51.923648 23249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a354dcae9dd4dab8240107c4958a624 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:53:51.923843 23263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92754a93811c4c748368ceaa258d80f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94f5d38aadaa4469ae7cfe8bedf10346. Latest consensus state: current_term: 1 leader_uuid: "94f5d38aadaa4469ae7cfe8bedf10346" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f5d38aadaa4469ae7cfe8bedf10346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43359 } } peers { permanent_uuid: "92754a93811c4c748368ceaa258d80f6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33413 } } peers { permanent_uuid: "9a354dcae9dd4dab8240107c4958a624" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39041 } } }
I20260619 16:53:51.923944 23263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92754a93811c4c748368ceaa258d80f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:53:51.924170 23299 mvcc.cc:205] Tried to move back new op lower bound from 7298613378736005120 to 7298613378685542400. Current Snapshot: MvccSnapshot[applied={T|T < 7298613378736005120}]
I20260619 16:53:51.924847 23301 mvcc.cc:205] Tried to move back new op lower bound from 7298613378736005120 to 7298613378685542400. Current Snapshot: MvccSnapshot[applied={T|T < 7298613378736005120}]
I20260619 16:53:51.925472 23297 catalog_manager.cc:1347] Generated new cluster ID: ced79082452141c6adbb65750a233484
I20260619 16:53:51.925519 23297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:53:51.930419 23277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94f5d38aadaa4469ae7cfe8bedf10346 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "94f5d38aadaa4469ae7cfe8bedf10346" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f5d38aadaa4469ae7cfe8bedf10346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43359 } } peers { permanent_uuid: "92754a93811c4c748368ceaa258d80f6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33413 } } peers { permanent_uuid: "9a354dcae9dd4dab8240107c4958a624" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39041 } } }
I20260619 16:53:51.930672 23277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94f5d38aadaa4469ae7cfe8bedf10346 [sys.catalog]: This master's current role is: LEADER
I20260619 16:53:51.931042 23263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92754a93811c4c748368ceaa258d80f6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "94f5d38aadaa4469ae7cfe8bedf10346" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f5d38aadaa4469ae7cfe8bedf10346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43359 } } peers { permanent_uuid: "92754a93811c4c748368ceaa258d80f6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33413 } } peers { permanent_uuid: "9a354dcae9dd4dab8240107c4958a624" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39041 } } }
I20260619 16:53:51.931154 23263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92754a93811c4c748368ceaa258d80f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:53:51.931336 23304 mvcc.cc:205] Tried to move back new op lower bound from 7298613378736005120 to 7298613378685542400. Current Snapshot: MvccSnapshot[applied={T|T < 7298613378736005120}]
I20260619 16:53:51.959532 23297 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:53:51.960573 23297 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:53:51.971832 23297 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94f5d38aadaa4469ae7cfe8bedf10346: Generated new TSK 0
I20260619 16:53:51.972283 23297 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 16:53:52.003295 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:53:52.000169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56626 (local address 127.0.98.61:33413)
0619 16:53:52.000230 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:52.000232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:52.002411 (+ 2179us) server_negotiation.cc:407] Connection header received
0619 16:53:52.002532 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:52.002534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:52.002550 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:52.002581 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:52.003192 (+ 611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56626: BlockingRecv error: recv got EOF from 127.0.0.1:56626 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260619 16:53:52.004288 23096 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:53:52.000715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57486 (local address 127.0.98.62:43359)
0619 16:53:52.000776 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:52.000779 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:52.003602 (+ 2823us) server_negotiation.cc:407] Connection header received
0619 16:53:52.003653 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:52.003655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:52.003671 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:52.003759 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:52.004202 (+ 443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57486: BlockingRecv error: recv got EOF from 127.0.0.1:57486 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260619 16:53:52.004405 23246 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:53:52.001116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33414 (local address 127.0.98.60:39041)
0619 16:53:52.001168 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:52.001170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:52.003746 (+ 2576us) server_negotiation.cc:407] Connection header received
0619 16:53:52.003842 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:52.003843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:52.003855 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:52.003886 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:52.004330 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33414: BlockingRecv error: recv got EOF from 127.0.0.1:33414 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260619 16:53:52.007666 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:53:52.006670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56628 (local address 127.0.98.61:33413)
0619 16:53:52.006722 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:52.006724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:52.006972 (+ 248us) server_negotiation.cc:407] Connection header received
0619 16:53:52.007131 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:52.007132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:52.007156 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:52.007188 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:52.007597 (+ 409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56628: BlockingRecv error: recv got EOF from 127.0.0.1:56628 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260619 16:53:52.011368 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:53:52.010398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56630 (local address 127.0.98.61:33413)
0619 16:53:52.010441 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:52.010442 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:52.010717 (+ 275us) server_negotiation.cc:407] Connection header received
0619 16:53:52.010821 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:52.010822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:52.010836 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:52.010868 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:52.011289 (+ 421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56630: BlockingRecv error: recv got EOF from 127.0.0.1:56630 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260619 16:53:52.015790 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:53:52.014828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56638 (local address 127.0.98.61:33413)
0619 16:53:52.014878 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:52.014880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:52.015161 (+ 281us) server_negotiation.cc:407] Connection header received
0619 16:53:52.015270 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:52.015271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:52.015286 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:52.015316 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:52.015722 (+ 406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56638: BlockingRecv error: recv got EOF from 127.0.0.1:56638 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260619 16:53:52.022253 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:53:52.021294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56648 (local address 127.0.98.61:33413)
0619 16:53:52.021340 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:52.021342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:52.021602 (+ 260us) server_negotiation.cc:407] Connection header received
0619 16:53:52.021699 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:52.021700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:52.021713 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:52.021740 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:52.022141 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56648: BlockingRecv error: recv got EOF from 127.0.0.1:56648 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260619 16:53:52.032856 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:53:52.031783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56652 (local address 127.0.98.61:33413)
0619 16:53:52.031847 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:52.031849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:52.032181 (+ 332us) server_negotiation.cc:407] Connection header received
0619 16:53:52.032291 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:52.032292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:52.032305 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:52.032336 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:52.032779 (+ 443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56652: BlockingRecv error: recv got EOF from 127.0.0.1:56652 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260619 16:53:52.051234 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:53:52.050409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56660 (local address 127.0.98.61:33413)
0619 16:53:52.050465 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:52.050467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:52.050624 (+ 157us) server_negotiation.cc:407] Connection header received
0619 16:53:52.050695 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:52.050696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:52.050710 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:52.050760 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:52.051161 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56660: BlockingRecv error: recv got EOF from 127.0.0.1:56660 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260619 16:53:52.085210 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:53:52.084448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56662 (local address 127.0.98.61:33413)
0619 16:53:52.084509 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:52.084512 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:52.084634 (+ 122us) server_negotiation.cc:407] Connection header received
0619 16:53:52.084708 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:52.084710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:52.084722 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:52.084766 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:52.085150 (+ 384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56662: BlockingRecv error: recv got EOF from 127.0.0.1:56662 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260619 16:53:52.151314 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:53:52.150628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56668 (local address 127.0.98.61:33413)
0619 16:53:52.150684 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:52.150686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:52.150763 (+ 77us) server_negotiation.cc:407] Connection header received
0619 16:53:52.150846 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:52.150847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:52.150858 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:52.150902 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:52.151250 (+ 348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56668: BlockingRecv error: recv got EOF from 127.0.0.1:56668 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260619 16:53:52.281560 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:53:52.280722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56684 (local address 127.0.98.61:33413)
0619 16:53:52.280834 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:52.280836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:52.280866 (+ 30us) server_negotiation.cc:407] Connection header received
0619 16:53:52.280983 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:52.280984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:52.281004 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:52.281060 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:52.281474 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56684: BlockingRecv error: recv got EOF from 127.0.0.1:56684 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260619 16:53:52.539824 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:53:52.539015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56690 (local address 127.0.98.61:33413)
0619 16:53:52.539092 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:52.539095 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:52.539131 (+ 36us) server_negotiation.cc:407] Connection header received
0619 16:53:52.539243 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:52.539245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:52.539257 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:52.539310 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:53:52.539751 (+ 441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56690: BlockingRecv error: recv got EOF from 127.0.0.1:56690 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 16:53:52.836354 23260 catalog_manager.cc:1259] Loaded cluster ID: ced79082452141c6adbb65750a233484
I20260619 16:53:52.836450 23260 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9a354dcae9dd4dab8240107c4958a624: loading cluster ID for follower catalog manager: success
I20260619 16:53:52.837296 23260 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9a354dcae9dd4dab8240107c4958a624: acquiring CA information for follower catalog manager: success
I20260619 16:53:52.837581 23260 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9a354dcae9dd4dab8240107c4958a624: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:53:52.871450 23274 catalog_manager.cc:1259] Loaded cluster ID: ced79082452141c6adbb65750a233484
I20260619 16:53:52.871551 23274 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 92754a93811c4c748368ceaa258d80f6: loading cluster ID for follower catalog manager: success
I20260619 16:53:52.872408 23274 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 92754a93811c4c748368ceaa258d80f6: acquiring CA information for follower catalog manager: success
I20260619 16:53:52.872709 23274 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 92754a93811c4c748368ceaa258d80f6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:53:53.056102 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:53.053285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56704 (local address 127.0.98.61:33413)
0619 16:53:53.053422 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:53.053424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:53.053440 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:53:53.053542 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:53.053543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:53.053618 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:53.053696 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:53:53.053859 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:53.054468 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:53.055269 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:53.055487 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:53.055546 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:53.055719 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:53:53.055860 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:53:53.055946 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:53.055963 (+ 17us) server_negotiation.cc:299] Negotiation successful
0619 16:53:53.055993 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":62,"threads_started":1}
I20260619 16:53:53.061316 23378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:53.058355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33428 (local address 127.0.98.60:39041)
0619 16:53:53.058561 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:53.058563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:53.058573 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:53:53.058662 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:53.058663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:53.058713 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:53.058796 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:53:53.058909 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:53.059440 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:53.060208 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:53.060420 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:53.060485 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:53.060880 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:53:53.061024 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:53:53.061137 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:53.061179 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 16:53:53.061209 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":60,"threads_started":1}
I20260619 16:53:53.066079 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:53.063330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56714 (local address 127.0.98.61:33413)
0619 16:53:53.063491 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:53.063493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:53.063502 (+ 9us) server_negotiation.cc:407] Connection header received
0619 16:53:53.063534 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:53.063536 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:53.063550 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:53.063602 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:53:53.063714 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:53.064482 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:53.065335 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:53.065553 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:53.065612 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:53.065788 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:53:53.065829 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:53:53.065878 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:53.065944 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 16:53:53.065978 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260619 16:53:53.066469 23378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:53.063503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33430 (local address 127.0.98.60:39041)
0619 16:53:53.063555 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:53.063557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:53.063744 (+ 187us) server_negotiation.cc:407] Connection header received
0619 16:53:53.063812 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:53.063813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:53.063824 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:53.063863 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:53:53.063997 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:53.064459 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:53.065545 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:53.065830 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:53.065883 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:53.066147 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:53:53.066212 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:53:53.066261 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:53.066356 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 16:53:53.066385 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 16:53:53.066998 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:53:53.063383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57502 (local address 127.0.98.62:43359)
0619 16:53:53.063589 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 16:53:53.063591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 16:53:53.063602 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:53:53.063656 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:53:53.063658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:53:53.063728 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:53:53.063816 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:53:53.063939 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:53.064690 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:53.065664 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:53:53.065924 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:53:53.065987 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:53:53.066450 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:53:53.066781 (+ 331us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:53:53.066845 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 16:53:53.066896 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 16:53:53.066928 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":60,"threads_started":1}
I20260619 16:53:53.068950 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 23025
I20260619 16:53:53.074398 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 23100
I20260619 16:53:53.079445 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjYBH_Y/build/release/bin/kudu with pid 23175
2026-06-19T16:53:53Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2897 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5720 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (291181 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST